arXiv subjects

Yuan Yao

Publications and source records attributed to Yuan Yao.

At least 253 records · Page 14Linked to original sources

arXiv2021

Generalization Bounds for Stochastic Gradient Langevin Dynamics: A Unified View via Information Leakage Analysis

Recently, generalization bounds of the non-convex empirical risk minimization paradigm using Stochastic Gradient Langevin Dynamics (SGLD) have been extensively studied. Several theoretical frameworks have been presented to study this problem from different perspectives, such as information theory and stability. In this paper, we present a unified view from privacy leakage analysis to investigate the generalization bounds of SGLD, along with a theoretical framework for re-deriving previous results in a succinct manner. Aside from theoretical findings, we conduct various numerical studies to empirically assess the information leakage issue of SGLD. Additionally, our theoretical and empirical results provide explanations for prior works that study the membership privacy of SGLD.

cs.LG↗

arXiv2021

Backdoor Attack through Frequency Domain

Backdoor attacks have been shown to be a serious threat against deep learning systems such as biometric authentication and autonomous driving. An effective backdoor attack could enforce the model misbehave under certain predefined conditions, i.e., triggers, but behave normally otherwise. However, the triggers of existing attacks are directly injected in the pixel space, which tend to be detectable by existing defenses and visually identifiable at both training and inference stages. In this paper, we propose a new backdoor attack FTROJAN through trojaning the frequency domain. The key intuition is that triggering perturbations in the frequency domain correspond to small pixel-wise perturbations dispersed across the entire image, breaking the underlying assumptions of existing defenses and making the poisoning images visually indistinguishable from clean ones. We evaluate FTROJAN in several datasets and tasks showing that it achieves a high attack success rate without significantly degrading the prediction accuracy on benign inputs. Moreover, the poisoning images are nearly invisible and retain high perceptual quality. We also evaluate FTROJAN against state-of-the-art defenses as well as several adaptive defenses that are designed on the frequency domain. The results show that FTROJAN can robustly elude or significantly degenerate the performance of these defenses.

cs.CR↗

arXiv2021

Early DSE and Automatic Generation of Coarse Grained Merged Accelerators

Post-Moore's law area-constrained systems rely on accelerators to deliver performance enhancements. Coarse grained accelerators can offer substantial domain acceleration, but manual, ad-hoc identification of code to accelerate is prohibitively expensive. Because cycle-accurate simulators and high-level synthesis flows are so time-consuming, manual creation of high-utilization accelerators that exploit control and data flow patterns at optimal granularities is rarely successful. To address these challenges, we present AccelMerger, the first automated methodology to create coarse grained, control- and data-flow-rich, merged accelerators. AccelMerger uses sequence alignment matching to recognize similar function call-graphs and loops, and neural networks to quickly evaluate their post-HLS characteristics. It accurately identifies which functions to accelerate, and it merges accelerators to respect an area budget and to accommodate system communication characteristics like latency and bandwidth. Merging two accelerators can save as much as 99% of the area of one. The space saved is used by a globally optimal integer linear program to allocate more accelerators for increased performance. We demonstate AccelMerger's effectiveness using HLS flows without any manual effort to fine-tune the resulting designs. On FPGA-based systems, AccelMerger yields application performance improvements of up to 16.7x over software implementations, and 1.91x on average with respect to state-of-the-art early-stage design space exploration tools.

cs.AR↗

arXiv2021

Accurate energies of transition metal atoms, ions, and monoxides using selected configuration interaction and density-based basis-set corrections

The semistochastic heat-bath configuration interaction (SHCI) method is a selected configuration interaction plus perturbation theory method that has provided near-full configuration interaction (FCI) levels of accuracy for many systems with both single- and multi-reference character. However, obtaining accurate energies in the complete basis set limit is hindered by the slow convergence of the FCI energy with respect to basis size. Here we show that the recently developed basis-set correction method based on range-separated density-functional theory can be used to significantly speed up basis-set convergence in SHCI calculations. In particular, we study two such schemes that differ in the functional used, and apply them to transition metal atoms and monoxides to obtain total, ionization, and dissociation energies well converged to the complete-basis-set limit within chemical accuracy.

physics.chem-ph↗

arXiv2021

ONION: A Simple and Effective Defense Against Textual Backdoor Attacks

Backdoor attacks are a kind of emergent training-time threat to deep neural networks (DNNs). They can manipulate the output of DNNs and possess high insidiousness. In the field of natural language processing, some attack methods have been proposed and achieve very high attack success rates on multiple popular models. Nevertheless, there are few studies on defending against textual backdoor attacks. In this paper, we propose a simple and effective textual backdoor defense named ONION, which is based on outlier word detection and, to the best of our knowledge, is the first method that can handle all the textual backdoor attack situations. Experiments demonstrate the effectiveness of our model in defending BiLSTM and BERT against five different backdoor attacks. All the code and data of this paper can be obtained at https://github.com/thunlp/ONION.

cs.CL↗

arXiv2021

The dynamic chirality flips of Skyrmion bubbles

Magnetic skyrmion, a topological magnetic domain with complex non-coplanar spin texture, appears a disk-like structure in two dimensions. Exploring three-dimensional spin texture and related chirality switching has drawn enormous interests from the perspective of fundamental research. Here, the three-dimensional magnetic moment of the skyrmion bubbles in centrosymmetric Mn-Ni-Ga were reconstructed with the vector field tomography approach via Lorentz transmission electron microscopy. The type of the bubbles was determined from investigating the magnetic vectors in entire space. We found that the bubbles switched their chirality easily but still keep the polarity to remain the singularity of the bubbles within the material. Our results offer valuable insights into the fundamental mechanisms underlying the spin chirality flips dynamics of skyrmion bubbles.

cond-mat.mtrl-sci↗

arXiv2021

Natural Gradient Optimization for Optical Quantum Circuits

Optical quantum circuits can be optimized using gradient descent methods, as the gates in a circuit can be parametrized by continuous parameters. However, the parameter space as seen by the cost function is not Euclidean, which means that the Euclidean gradient does not generally point in the direction of steepest ascent. In order to retrieve the steepest ascent direction, in this work we implement Natural Gradient descent in the optical quantum circuit setting, which takes the local metric tensor into account. In particular, we adapt the Natural Gradient approach to a complex-valued parameter space. We then compare the Natural Gradient approach to vanilla gradient descent and to Adam over two state preparation tasks: a single-photon source and a Gottesman-Kitaev-Preskill state source. We observe that the NG approach has a faster convergence (due in part to the possibility of using larger learning rates) and a significantly smoother decay of the cost function throughout the optimization.

quant-ph↗

arXiv2021

Federated Deep Learning with Bayesian Privacy

Federated learning (FL) aims to protect data privacy by cooperatively learning a model without sharing private data among users. For Federated Learning of Deep Neural Network with billions of model parameters, existing privacy-preserving solutions are unsatisfactory. Homomorphic encryption (HE) based methods provide secure privacy protections but suffer from extremely high computational and communication overheads rendering it almost useless in practice . Deep learning with Differential Privacy (DP) was implemented as a practical learning algorithm at a manageable cost in complexity. However, DP is vulnerable to aggressive Bayesian restoration attacks as disclosed in the literature and demonstrated in experimental results of this work. To address the aforementioned perplexity, we propose a novel Bayesian Privacy (BP) framework which enables Bayesian restoration attacks to be formulated as the probability of reconstructing private data from observed public information. Specifically, the proposed BP framework accurately quantifies privacy loss by Kullback-Leibler (KL) Divergence between the prior distribution about the privacy data and the posterior distribution of restoration private data conditioning on exposed information}. To our best knowledge, this Bayesian Privacy analysis is the first to provides theoretical justification of secure privacy-preserving capabilities against Bayesian restoration attacks. As a concrete use case, we demonstrate that a novel federated deep learning method using private passport layers is able to simultaneously achieve high model performance, privacy-preserving capability and low computational complexity. Theoretical analysis is in accordance with empirical measurements of information leakage extensively experimented with a variety of DNN networks on image classification MNIST, CIFAR10, and CIFAR100 datasets.

cs.LG↗

arXiv2021

On ADMM in Deep Learning: Convergence and Saturation-Avoidance

In this paper, we develop an alternating direction method of multipliers (ADMM) for deep neural networks training with sigmoid-type activation functions (called \textit{sigmoid-ADMM pair}), mainly motivated by the gradient-free nature of ADMM in avoiding the saturation of sigmoid-type activations and the advantages of deep neural networks with sigmoid-type activations (called deep sigmoid nets) over their rectified linear unit (ReLU) counterparts (called deep ReLU nets) in terms of approximation. In particular, we prove that the approximation capability of deep sigmoid nets is not worse than that of deep ReLU nets by showing that ReLU activation function can be well approximated by deep sigmoid nets with two hidden layers and finitely many free parameters but not vice-verse. We also establish the global convergence of the proposed ADMM for the nonlinearly constrained formulation of the deep sigmoid nets training from arbitrary initial points to a Karush-Kuhn-Tucker (KKT) point at a rate of order ${\cal O}(1/k)$. Besides sigmoid activation, such a convergence theorem holds for a general class of smooth activations. Compared with the widely used stochastic gradient descent (SGD) algorithm for the deep ReLU nets training (called ReLU-SGD pair), the proposed sigmoid-ADMM pair is practically stable with respect to the algorithmic hyperparameters including the learning rate, initial schemes and the pro-processing of the input data. Moreover, we find that to approximate and learn simple but important functions the proposed sigmoid-ADMM pair numerically outperforms the ReLU-SGD pair.

cs.LG↗

arXiv2021

Multi-source Heterogeneous Domain Adaptation with Conditional Weighting Adversarial Network

Heterogeneous domain adaptation (HDA) tackles the learning of cross-domain samples with both different probability distributions and feature representations. Most of the existing HDA studies focus on the single-source scenario. In reality, however, it is not uncommon to obtain samples from multiple heterogeneous domains. In this article, we study the multisource HDA problem and propose a conditional weighting adversarial network (CWAN) to address it. The proposed CWAN adversarially learns a feature transformer, a label classifier, and a domain discriminator. To quantify the importance of different source domains, CWAN introduces a sophisticated conditional weighting scheme to calculate the weights of the source domains according to the conditional distribution divergence between the source and target domains. Different from existing weighting schemes, the proposed conditional weighting scheme not only weights the source domains but also implicitly aligns the conditional distributions during the optimization process. Experimental results clearly demonstrate that the proposed CWAN performs much better than several state-of-the-art methods on four real-world datasets.

cs.LG↗

arXiv2021

Differentially Private Federated Learning with Laplacian Smoothing

Federated learning aims to protect data privacy by collaboratively learning a model without sharing private data among users. However, an adversary may still be able to infer the private training data by attacking the released model. Differential privacy provides a statistical protection against such attacks at the price of significantly degrading the accuracy or utility of the trained models. In this paper, we investigate a utility enhancement scheme based on Laplacian smoothing for differentially private federated learning (DP-Fed-LS), where the parameter aggregation with injected Gaussian noise is improved in statistical precision without losing privacy budget. Our key observation is that the aggregated gradients in federated learning often enjoy a type of smoothness, i.e. sparsity in the graph Fourier basis with polynomial decays of Fourier coefficients as frequency grows, which can be exploited by the Laplacian smoothing efficiently. Under a prescribed differential privacy budget, convergence error bounds with tight rates are provided for DP-Fed-LS with uniform subsampling of heterogeneous Non-IID data, revealing possible utility improvement of Laplacian smoothing in effective dimensionality and variance reduction, among others. Experiments over MNIST, SVHN, and Shakespeare datasets show that the proposed method can improve model accuracy with DP-guarantee and membership privacy under both uniform and Poisson subsampling mechanisms.

cs.LG↗

arXiv2021

Coherent spin rotation-induced zero thermal expansion in MnCoSi-based spiral magnets

Materials exhibiting zero thermal expansion (ZTE), namely, volume invariance during temperature change, can resist thermal shock and are highly desired in modern industries as high-precision components. However, pure ZTE materials are rare, especially those that are metallic. Here, we report the discovery of a pure metallic ZTE material: an orthorhombic Mn1-xNixCoSi spiral magnet. The introduction of Ni can efficiently enhance the ferromagnetic exchange interaction and construct the transition from a spiral magnetic state to a ferromagnetic-like state in MnCoSi-based alloys. Systematic in situ neutron powder diffraction revealed a new cycloidal spiral magnetic structure in bc plane at ground state which would transform to the helical spiral in the ab plane with increasing temperature. Combined with Lorentz transmission electron microscopy techniques, the cycloidal and helical spin order coherently rotated at varying periods along the c axis during the magnetic transition. This spin rotation drove the continuous movement of the coupled crystalline lattice and induced a large negative thermal expansion along the a axis, eventually leading to a wide-temperature ZTE effect. Our work not only introduces a new ZTE alloy but also presents a new mechanism by which to discover or design ZTE magnets.

cond-mat.mtrl-sci↗

arXiv2021

Dimming Down LED: An Open-source Threshold Implementation on Light Encryption Device (LED) Block Cipher

Lightweight block ciphers have been widely used in applications such as RFID tags, IoTs, and network sensors. Among them, with comparable parameters, the Light Encryption Device (LED) block cipher achieves the smallest area. However, implementation of encryption algorithms manifest side-channel leakage, therefore, it is crucial to protect their design against side-channel analyses. In this paper, we present a threshold implementation of the LED cipher which has 64-bit data input and 128-bit key. The presented design splits secret information among multiple shares to achieve a higher security level. We demonstrate that our implementation can protect against first-order power side-channel attacks. As a cost, the design area is almost doubled and the maximum operating frequency is degraded by 30%. To make our design verifiable, we have also open-sourced our design online.

cs.CR↗

arXiv2021

Visual Distant Supervision for Scene Graph Generation

Scene graph generation aims to identify objects and their relations in images, providing structured image representations that can facilitate numerous applications in computer vision. However, scene graph models usually require supervised learning on large quantities of labeled data with intensive human annotation. In this work, we propose visual distant supervision, a novel paradigm of visual relation learning, which can train scene graph models without any human-labeled data. The intuition is that by aligning commonsense knowledge bases and images, we can automatically create large-scale labeled data to provide distant supervision for visual relation learning. To alleviate the noise in distantly labeled data, we further propose a framework that iteratively estimates the probabilistic relation labels and eliminates the noisy ones. Comprehensive experimental results show that our distantly supervised model outperforms strong weakly supervised and semi-supervised baselines. By further incorporating human-labeled data in a semi-supervised fashion, our model outperforms state-of-the-art fully supervised models by a large margin (e.g., 8.3 micro- and 7.8 macro-recall@50 improvements for predicate classification in Visual Genome evaluation). We make the data and code for this paper publicly available at https://github.com/thunlp/VisualDS.

cs.CV↗

arXiv2021

Pre-Trained Models: Past, Present and Future

Large-scale pre-trained models (PTMs) such as BERT and GPT have recently achieved great success and become a milestone in the field of artificial intelligence (AI). Owing to sophisticated pre-training objectives and huge model parameters, large-scale PTMs can effectively capture knowledge from massive labeled and unlabeled data. By storing knowledge into huge parameters and fine-tuning on specific tasks, the rich knowledge implicitly encoded in huge parameters can benefit a variety of downstream tasks, which has been extensively demonstrated via experimental verification and empirical analysis. It is now the consensus of the AI community to adopt PTMs as backbone for downstream tasks rather than learning models from scratch. In this paper, we take a deep look into the history of pre-training, especially its special relation with transfer learning and self-supervised learning, to reveal the crucial position of PTMs in the AI development spectrum. Further, we comprehensively review the latest breakthroughs of PTMs. These breakthroughs are driven by the surge of computational power and the increasing availability of data, towards four important directions: designing effective architectures, utilizing rich contexts, improving computational efficiency, and conducting interpretation and theoretical analysis. Finally, we discuss a series of open problems and research directions of PTMs, and hope our view can inspire and advance the future study of PTMs.

cs.AI↗

arXiv2021

Magnetic anisotropy and critical behavior of the quaternary van der Waals ferromagnetic material $\bf CrGe_δSi_{1-δ}Te_3$

Recently, two-dimensional ferromagnetism in the family of Chromium compounds $\rm CrXTe_3 (X=Si, Ge)$ has attracted a broad research interest. Despite the structural similarity in $\rm CrTe_6$ octahedra, the size effect of inserted Ge or Si dimer contributes to significant differences in magnetism. Here, we report a new quaternary van der Waals ferromagnetic material $\rm CrGe_δSi_{1-δ}Te_3$ synthesized by flux method. Ge substitution in Si site results in the lattice expansion, further increasing the Curie temperature and reducing the magnetic anisotropy. The critical behavior of $\rm Cr_{0.96}Ge_{0.17}Si_{0.82}Te_3$ has been studied by specific heat as well as magnetization measurements. And the extracted critical exponents are self-consistent and well-obeying the scaling laws, which are closer to the 2D Ising model with interaction decaying as $J(r)\approx r^{-3.44}$.

physics.app-ph↗

arXiv2021

A Note on Learning Rare Events in Molecular Dynamics using LSTM and Transformer

Recurrent neural networks for language models like long short-term memory (LSTM) have been utilized as a tool for modeling and predicting long term dynamics of complex stochastic molecular systems. Recently successful examples on learning slow dynamics by LSTM are given with simulation data of low dimensional reaction coordinate. However, in this report we show that the following three key factors significantly affect the performance of language model learning, namely dimensionality of reaction coordinates, temporal resolution and state partition. When applying recurrent neural networks to molecular dynamics simulation trajectories of high dimensionality, we find that rare events corresponding to the slow dynamics might be obscured by other faster dynamics of the system, and cannot be efficiently learned. Under such conditions, we find that coarse graining the conformational space into metastable states and removing recrossing events when estimating transition probabilities between states could greatly help improve the accuracy of slow dynamics learning in molecular dynamics. Moreover, we also explore other models like Transformer, which do not show superior performance than LSTM in overcoming these issues. Therefore, to learn rare events of slow molecular dynamics by LSTM and Transformer, it is critical to choose proper temporal resolution (i.e., saving intervals of MD simulation trajectories) and state partition in high resolution data, since deep neural network models might not automatically disentangle slow dynamics from fast dynamics when both are present in data influencing each other.

cs.AI↗

arXiv2021

Real-time Detection and Adaptive Mitigation of Power-based Side-Channel Leakage in SoC

Power-based side-channel is a serious security threat to the System on Chip (SoC). The secret information is leaked from the power profile of the system while a cryptographic algorithm is running. The mitigation requires efforts from both the software level and hardware level. Currently, there is no comprehensive solution that can guarantee the whole complex system is free of leakage and can generically protect all cryptographic algorithms. In this paper, we propose a real-time leakage detection and mitigation system which enables the system to monitor the side-channel leakage effects of the hardware. Our proposed system has extensions that provide a real-time monitor of power consumption, detection of side-channel leakage, and real-time adaptive mitigation of detected side-channel leakage. Our proposed system is generic and can protect any algorithm running on it.

cs.CR↗