arXiv subjects

Hai Li

Publications and source records attributed to Hai Li.

At least 109 records · Page 6Linked to original sources

arXiv2022

Approximate Computing and the Efficient Machine Learning Expedition

Approximate computing (AxC) has been long accepted as a design alternative for efficient system implementation at the cost of relaxed accuracy requirements. Despite the AxC research activities in various application domains, AxC thrived the past decade when it was applied in Machine Learning (ML). The by definition approximate notion of ML models but also the increased computational overheads associated with ML applications-that were effectively mitigated by corresponding approximations-led to a perfect matching and a fruitful synergy. AxC for AI/ML has transcended beyond academic prototypes. In this work, we enlighten the synergistic nature of AxC and ML and elucidate the impact of AxC in designing efficient ML systems. To that end, we present an overview and taxonomy of AxC for ML and use two descriptive application scenarios to demonstrate how AxC boosts the efficiency of ML systems.

cs.AR↗

arXiv2022

NashAE: Disentangling Representations through Adversarial Covariance Minimization

We present a self-supervised method to disentangle factors of variation in high-dimensional data that does not rely on prior knowledge of the underlying variation profile (e.g., no assumptions on the number or distribution of the individual latent variables to be extracted). In this method which we call NashAE, high-dimensional feature disentanglement is accomplished in the low-dimensional latent space of a standard autoencoder (AE) by promoting the discrepancy between each encoding element and information of the element recovered from all other encoding elements. Disentanglement is promoted efficiently by framing this as a minmax game between the AE and an ensemble of regression networks which each provide an estimate of an element conditioned on an observation of all other elements. We quantitatively compare our approach with leading disentanglement methods using existing disentanglement metrics. Furthermore, we show that NashAE has increased reliability and increased capacity to capture salient data characteristics in the learned latent representation.

cs.LG↗

arXiv2022

Fine-grain Inference on Out-of-Distribution Data with Hierarchical Classification

Machine learning methods must be trusted to make appropriate decisions in real-world environments, even when faced with out-of-distribution (OOD) samples. Many current approaches simply aim to detect OOD examples and alert the user when an unrecognized input is given. However, when the OOD sample significantly overlaps with the training data, a binary anomaly detection is not interpretable or explainable, and provides little information to the user. We propose a new model for OOD detection that makes predictions at varying levels of granularity as the inputs become more ambiguous, the model predictions become coarser and more conservative. Consider an animal classifier that encounters an unknown bird species and a car. Both cases are OOD, but the user gains more information if the classifier recognizes that its uncertainty over the particular species is too large and predicts bird instead of detecting it as OOD. Furthermore, we diagnose the classifiers performance at each level of the hierarchy improving the explainability and interpretability of the models predictions. We demonstrate the effectiveness of hierarchical classifiers for both fine- and coarse-grained OOD tasks.

cs.LG↗

arXiv2022

Joint Optimization of STAR-RIS Assisted UAV Communication Systems

In this letter, we study the simultaneously transmitting and reflecting reconfigurable intelligent surface (STAR-RIS) assisted unmanned aerial vehicle (UAV) communications. Our goal is to maximize the sum rate of all users by jointly optimizing the STAR-RIS's beamforming vectors, the UAV's trajectory and power allocation. We decompose the formulated non-convex problem into three subproblems and solve them alternately to obtain the solution. Simulations show that: 1) the STAR-RIS achieves a higher sum rate than traditional RIS; 2) to exploit the benefits of STAR-RIS, the UAV's trajectory is closer to STAR-RIS than that of RIS; 3) the energy splitting for reflection and transmission highly depends on the real-time trajectory of UAV.

cs.IT↗

arXiv2022

Learning and Compositionality: a Unification Attempt via Connectionist Probabilistic Programming

We consider learning and compositionality as the key mechanisms towards simulating human-like intelligence. While each mechanism is successfully achieved by neural networks and symbolic AIs, respectively, it is the combination of the two mechanisms that makes human-like intelligence possible. Despite the numerous attempts on building hybrid neuralsymbolic systems, we argue that our true goal should be unifying learning and compositionality, the core mechanisms, instead of neural and symbolic methods, the surface approaches to achieve them. In this work, we review and analyze the strengths and weaknesses of neural and symbolic methods by separating their forms and meanings (structures and semantics), and propose Connectionist Probabilistic Program (CPPs), a framework that connects connectionist structures (for learning) and probabilistic program semantics (for compositionality). Under the framework, we design a CPP extension for small scale sequence modeling and provide a learning algorithm based on Bayesian inference. Although challenges exist in learning complex patterns without supervision, our early results demonstrate CPP's successful extraction of concepts and relations from raw sequential data, an initial step towards compositional learning.

cs.AI↗

arXiv2022

Nonvolatile Electric Field Control of Thermal Magnons in the Absence of an Applied Magnetic Field

Spin transport through magnetic insulators has been demonstrated in a variety of materials and is an emerging pathway for next-generation spin-based computing. To modulate spin transport in these systems, one typically applies a sufficiently strong magnetic field to allow for deterministic control of magnetic order. Here, we make use of the well-known multiferroic magnetoelectric, BiFeO3, to demonstrate non-volatile, hysteretic, electric-field control of thermally excited magnon current in the absence of an applied magnetic field. These findings are an important step toward magnon-based devices, where electric-field-only control is highly desirable.

cond-mat.mtrl-sci↗

arXiv2022

Spin Torque Generated by Valley Hall Effect in WSe2

Monolayer transition metal dichalcogenides are promising materials for spintronics due to their robust spin-valley locked valence states, enabling efficient charge-to-spin conversion via valley Hall effect with non-equilibrium spins possessing long spin diffusion lengths of hundreds of nanometers. In this work, we show that the injection of a pure valley current, induced by valley Hall effect in a WSe2 monolayer, imparts a spin torque on the magnetization of an overlaid Fe or CoFe in a tunneling structure. The torque efficiency is found to be comparable to that in conventional perpendicular magnetic tunnel junctions and can be further optimized with valley Hall angle in WSe2. The valley nature of the spin torque gives rise to out-of-plane damping-like torques in a current-in-plane configuration, vanishing charge transport perpendicular-to-the-plane as well as torque efficiency tunable through gating.

cond-mat.mes-hall↗

arXiv2022

Relevance between Information scrambling and quantum Darwinism

Quantum system interacting with environment can induce redundant encoding of the information of system into a multipartite environment, which is the essence of quantum Darwinism. At the same time, environment may scramble the initially localized information about the system. We mainly investigate the relevance between information scrambling in environment and the emergence of quantum Darwinism. First, we generally identify that when the system shows a Darwinistic behavior system information that is initially localized in the environment is not scrambled, while when Darwinism disappears scrambling occurs.We then verify our result through a collision model where the system, consisting of one or two qubits, interacts with an ensemble of environmental ancillas.Moreover, dependent on the nature of system-environment interactions, our results also shows that the single qubit and two-qubit systems behave differently for the emergence of QD and the scrambling, but the above relevance between them remains valid.

quant-ph↗

arXiv2022

Quantum information scrambling in non-Markovian open quantum systems

In this paper we investigate the dynamics of a spin chain whose two end spins interact with two independent non-Markovian baths by using the non-Markovian quantum state diffusion (QSD) equation approach. Specifically two issues about quantum information scrambling in open quantum system are addressed. The first issue is that tripartite mutual information (TMI) can quantify information scrambling properly via its negative value in closed system, whether it is still suitable to indicate quantum scrambling in open quantum system. However we find that negative TMI is not an suitable quantifier of information scrambling in open quantum system in some cases while negative tripartite logarithmic negativity (TLN) is more appropriate. The second one is that up to now almost all the open quantum system effects on information scrambling reported were focus on the Markovian environment, while the effect of non-Markovian environment on information scrambling is still elusive. Significantly our results show that the memory effect of environment will be beneficial to the emergence of quantum information scrambling. Moreover, it is found that environment is generally detrimental for information scrambling in a long time, while in some cases it will be helpful for information scrambling in a short time.

quant-ph↗

arXiv2022

First-principles Landau-like potential for BiFeO$_3$ and related materials

In this work we introduce the simplest, lowest-order Landau-like potential for BiFeO$_3$ and La-doped BiFeO$_3$ as an expansion around the paraelectric cubic phase in powers of polarization, FeO$_6$ octahedral rotations and strains. We present an analytical approach for computing the model parameters from density functional theory. We illustrate our approach by computing the potentials for BiFeO$_3$ and La$_{0.25}$Bi$_{0.75}$FeO$_3$ and show that, overall, we are able to capture the first-principles results accurately. The computed models allow us to identify and explain the main interactions controlling the relative stability of the competing low-energy phases of these compounds.

cond-mat.mtrl-sci↗

arXiv2022

FedCor: Correlation-Based Active Client Selection Strategy for Heterogeneous Federated Learning

Client-wise data heterogeneity is one of the major issues that hinder effective training in federated learning (FL). Since the data distribution on each client may vary dramatically, the client selection strategy can significantly influence the convergence rate of the FL process. Active client selection strategies are popularly proposed in recent studies. However, they neglect the loss correlations between the clients and achieve only marginal improvement compared to the uniform selection strategy. In this work, we propose FedCor -- an FL framework built on a correlation-based client selection strategy, to boost the convergence rate of FL. Specifically, we first model the loss correlations between the clients with a Gaussian Process (GP). Based on the GP model, we derive a client selection strategy with a significant reduction of expected global loss in each round. Besides, we develop an efficient GP training method with a low communication overhead in the FL scenario by utilizing the covariance stationarity. Our experimental results show that compared to the state-of-the-art method, FedCorr can improve the convergence rates by $34\%\sim 99\%$ and $26\%\sim 51\%$ on FMNIST and CIFAR-10, respectively.

cs.LG↗

arXiv2022

ReaLPrune: ReRAM Crossbar-aware Lottery Ticket Pruned CNNs

Training machine learning (ML) models at the edge (on-chip training on end user devices) can address many pressing challenges including data privacy/security, increase the accessibility of ML applications to different parts of the world by reducing the dependence on the communication fabric and the cloud infrastructure, and meet the real-time requirements of AR/VR applications. However, existing edge platforms do not have sufficient computing capabilities to support complex ML tasks such as training large CNNs. ReRAM-based architectures offer high-performance yet energy efficient computing platforms for on-chip CNN training/inferencing. However, ReRAM-based architectures are not scalable with the size of the CNN. Larger CNNs have more weights, which requires more ReRAM cells that cannot be integrated in a single chip. Moreover, training larger CNNs on-chip will require higher power, which cannot be afforded by these smaller devices. Pruning is an effective way to solve this problem. However, existing pruning techniques are either targeted for inferencing only, or they are not crossbar-aware. This leads to sub-optimal hardware savings and performance benefits for CNN training on ReRAM-based architectures. In this paper, we address this problem by proposing a novel crossbar-aware pruning strategy, referred as ReaLPrune, which can prune more than 90% of CNN weights. The pruned model can be trained from scratch without any accuracy loss. Experimental results indicate that ReaLPrune reduces hardware requirements by 77.2% and accelerates CNN training by ~20X compared to unpruned CNNs. ReaLPrune also outperforms other crossbar-aware pruning techniques in terms of both performance and hardware savings. In addition, ReaLPrune is equally effective for diverse datasets and more complex CNNs

cs.AR↗

arXiv2022

Field free switching through bulk spin-orbit torque in L10-FePt films deposited on vicinal substrates

L10-FePt distinguishes itself for its ultrahigh perpendicular magnetic anisotropy (PMA), which enables memory cells with sufficient thermal stability to scale down to 3 nm. The recently discovered "bulk" spin-orbit torques in L10-FePt provide an efficient and scalable way to manipulate the L10-FePt magnetization. However, the existence of external field during the switching limits its practical application, and therefore field-free switching of the L10-FePt is in highly demand. In this manuscript, we demonstrate the field-free switching of the L10-FePt by growing it on vicinal MgO (001) substrates. This method is different from previously established strategies, as it does not need to add other functional layers or create asymmetry in the film structure. We demonstrate the field-free switching is robust and can withstand strong field disturbance up to ~1 kOe. The dependence on vicinal angle, film thickness, and growth temperature demonstrated a wide operation window for the field-free switching of the L10-FePt. We confirmed that the physical origin of the field-free switching is the vicinal surface-induced the tilted anisotropy of L10-FePt. We quantitatively characterize the spin-orbit torques in the L10-FePt films, and found the spin-orbit torques are not significantly influenced by the lattice strain from vicinal substrates. Our results extend beyond the established strategies to realize field-free switching, and potentially could be applied to other magnetic and antiferromagnetic systems.

cond-mat.mtrl-sci↗

arXiv2022

Privacy Leakage of Adversarial Training Models in Federated Learning Systems

Adversarial Training (AT) is crucial for obtaining deep neural networks that are robust to adversarial attacks, yet recent works found that it could also make models more vulnerable to privacy attacks. In this work, we further reveal this unsettling property of AT by designing a novel privacy attack that is practically applicable to the privacy-sensitive Federated Learning (FL) systems. Using our method, the attacker can exploit AT models in the FL system to accurately reconstruct users' private training images even when the training batch size is large. Code is available at https://github.com/zjysteven/PrivayAttack_AT_FL.

cs.LG↗

arXiv2022

IQDUBBING: Prosody modeling based on discrete self-supervised speech representation for expressive voice conversion

Prosody modeling is important, but still challenging in expressive voice conversion. As prosody is difficult to model, and other factors, e.g., speaker, environment and content, which are entangled with prosody in speech, should be removed in prosody modeling. In this paper, we present IQDubbing to solve this problem for expressive voice conversion. To model prosody, we leverage the recent advances in discrete self-supervised speech representation (DSSR). Specifically, prosody vector is first extracted from pre-trained VQ-Wav2Vec model, where rich prosody information is embedded while most speaker and environment information are removed effectively by quantization. To further filter out the redundant information except prosody, such as content and partial speaker information, we propose two kinds of prosody filters to sample prosody from the prosody vector. Experiments show that IQDubbing is superior to baseline and comparison systems in terms of speech quality while maintaining prosody consistency and speaker similarity.

eess.AS↗

arXiv2021

Gaussian Random Number Generator with Reconfigurable Mean and Variance using Stochastic Magnetic Tunnel Junctions

Generating high-quality random numbers with a Gaussian probability distribution function is an important and resource consuming computational task for many applications in the fields of machine learning and Monte Carlo algorithms. Recently, CMOS-based digital hardware architectures have been explored as specialized Gaussian random number generators (GRNGs). These CMOS-based GRNGs have a large area and require entropy sources at their input which increase the computing cost. Here, we propose a GRNG that works on the principle of the Boltzmann law in a physical system made from an interconnected network of thermally unstable magnetic tunnel junctions. The proposed hardware can produce multi-bit Gaussian random numbers at a gigahertz speed and can be configured to generate distributions with a desired mean and variance. An analytical derivation of the required interconnection and bias strengths is provided followed by numerical simulations to demonstrate the functionalities of the GRNG.

cs.ET↗

arXiv2021

FL-WBC: Enhancing Robustness against Model Poisoning Attacks in Federated Learning from a Client Perspective

Federated learning (FL) is a popular distributed learning framework that trains a global model through iterative communications between a central server and edge devices. Recent works have demonstrated that FL is vulnerable to model poisoning attacks. Several server-based defense approaches (e.g. robust aggregation), have been proposed to mitigate such attacks. However, we empirically show that under extremely strong attacks, these defensive methods fail to guarantee the robustness of FL. More importantly, we observe that as long as the global model is polluted, the impact of attacks on the global model will remain in subsequent rounds even if there are no subsequent attacks. In this work, we propose a client-based defense, named White Blood Cell for Federated Learning (FL-WBC), which can mitigate model poisoning attacks that have already polluted the global model. The key idea of FL-WBC is to identify the parameter space where long-lasting attack effect on parameters resides and perturb that space during local training. Furthermore, we derive a certified robustness guarantee against model poisoning attacks and a convergence guarantee to FedAvg after applying our FL-WBC. We conduct experiments on FasionMNIST and CIFAR10 to evaluate the defense against state-of-the-art model poisoning attacks. The results demonstrate that our method can effectively mitigate model poisoning attack impact on the global model within 5 communication rounds with nearly no accuracy drop under both IID and Non-IID settings. Our defense is also complementary to existing server-based robust aggregation approaches and can further improve the robustness of FL under extremely strong attacks.

cs.LG↗

arXiv2021

A Layer-wise Adversarial-aware Quantization Optimization for Improving Robustness

Neural networks are getting better accuracy with higher energy and computational cost. After quantization, the cost can be greatly saved, and the quantized models are more hardware friendly with acceptable accuracy loss. On the other hand, recent research has found that neural networks are vulnerable to adversarial attacks, and the robustness of a neural network model can only be improved with defense methods, such as adversarial training. In this work, we find that adversarially-trained neural networks are more vulnerable to quantization loss than plain models. To minimize both the adversarial and the quantization losses simultaneously and to make the quantized model robust, we propose a layer-wise adversarial-aware quantization method, using the Lipschitz constant to choose the best quantization parameter settings for a neural network. We theoretically derive the losses and prove the consistency of our metric selection. The experiment results show that our method can effectively and efficiently improve the robustness of quantized adversarially-trained neural networks.

cs.LG↗