arXiv subjects

Hai Li

Publications and source records attributed to Hai Li.

At least 181 records · Page 10Linked to original sources

arXiv2018

Effect of coherence of nonthermal reservoirs on heat transport in a microscopic collision model

We investigate the heat transport between two nonthermal reservoirs based on a microscopic collision model. We consider a bipartite system consisting of two identical subsystems, and each subsystem interacts with its own local reservoir, which consists of a large collection of initially uncorrelated ancillas. Then a heat transport is formed between two reservoirs by a sequence of pairwise collisions (inter-subsystem and subsystem-local reservoir). In this paper we consider two kinds of reservoir's initial states, the thermal state, and the state with coherence whose diagonal elements are the same as that of the thermal state and the off-diaganal elements are nonzero. In this way, we define the effective temperature of the reservoir with coherence according to its diagonal elements. We find that for two reservoirs having coherence the direction of the steady current of heat is different for different phase differences between the two initial states of two reservoirs, especially the heat can transfer from the "cold reservoir" to the "hot reservoir" in the steady regime for particular phase difference. And in the limit of the effective temperature difference between the two reservoirs $ΔT\rightarrow0$, for most of the phase differences, the steady heat current increases with the increase of effective temperature until to the high effective temperature limit; while for the thermal state or particular phase difference the steady heat current decreases with the increase of temperature at high temperatures, and in this case the conductance can be obtained.

quant-ph↗

arXiv2018

Learning Intrinsic Sparse Structures within Long Short-Term Memory

Model compression is significant for the wide adoption of Recurrent Neural Networks (RNNs) in both user devices possessing limited resources and business clusters requiring quick responses to large-scale service requests. This work aims to learn structurally-sparse Long Short-Term Memory (LSTM) by reducing the sizes of basic structures within LSTM units, including input updates, gates, hidden states, cell states and outputs. Independently reducing the sizes of basic structures can result in inconsistent dimensions among them, and consequently, end up with invalid LSTM units. To overcome the problem, we propose Intrinsic Sparse Structures (ISS) in LSTMs. Removing a component of ISS will simultaneously decrease the sizes of all basic structures by one and thereby always maintain the dimension consistency. By learning ISS within LSTM units, the obtained LSTMs remain regular while having much smaller basic structures. Based on group Lasso regularization, our method achieves 10.59x speedup without losing any perplexity of a language modeling of Penn TreeBank dataset. It is also successfully evaluated through a compact model with only 2.69M weights for machine Question Answering of SQuAD dataset. Our approach is successfully extended to non- LSTM RNNs, like Recurrent Highway Networks (RHNs). Our source code is publicly available at https://github.com/wenwei202/iss-rnns

cs.LG↗

arXiv2017

TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning

High network communication cost for synchronizing gradients and parameters is the well-known bottleneck of distributed training. In this work, we propose TernGrad that uses ternary gradients to accelerate distributed deep learning in data parallelism. Our approach requires only three numerical levels {-1,0,1}, which can aggressively reduce the communication time. We mathematically prove the convergence of TernGrad under the assumption of a bound on gradients. Guided by the bound, we propose layer-wise ternarizing and gradient clipping to improve its convergence. Our experiments show that applying TernGrad on AlexNet does not incur any accuracy loss and can even improve accuracy. The accuracy loss of GoogLeNet induced by TernGrad is less than 2% on average. Finally, a performance model is proposed to study the scalability of TernGrad. Experiments show significant speed gains for various deep neural networks. Our source code is available.

cs.LG↗

arXiv2017

GraphR: Accelerating Graph Processing Using ReRAM

This paper presents GRAPHR, the first ReRAM-based graph processing accelerator. GRAPHR follows the principle of near-data processing and explores the opportunity of performing massive parallel analog operations with low hardware and energy cost. The analog computation is suit- able for graph processing because: 1) The algorithms are iterative and could inherently tolerate the imprecision; 2) Both probability calculation (e.g., PageRank and Collaborative Filtering) and typical graph algorithms involving integers (e.g., BFS/SSSP) are resilient to errors. The key insight of GRAPHR is that if a vertex program of a graph algorithm can be expressed in sparse matrix vector multiplication (SpMV), it can be efficiently performed by ReRAM crossbar. We show that this assumption is generally true for a large set of graph algorithms. GRAPHR is a novel accelerator architecture consisting of two components: memory ReRAM and graph engine (GE). The core graph computations are performed in sparse matrix format in GEs (ReRAM crossbars). The vector/matrix-based graph computation is not new, but ReRAM offers the unique opportunity to realize the massive parallelism with unprecedented energy efficiency and low hardware cost. With small subgraphs processed by GEs, the gain of performing parallel operations overshadows the wastes due to sparsity. The experiment results show that GRAPHR achieves a 16.01x (up to 132.67x) speedup and a 33.82x energy saving on geometric mean compared to a CPU baseline system. Com- pared to GPU, GRAPHR achieves 1.69x to 2.19x speedup and consumes 4.77x to 8.91x less energy. GRAPHR gains a speedup of 1.16x to 4.12x, and is 3.67x to 10.96x more energy efficiency compared to PIM-based architecture.

cs.DC↗

arXiv2017

Faster CNNs with Direct Sparse Convolutions and Guided Pruning

Phenomenally successful in practical inference problems, convolutional neural networks (CNN) are widely deployed in mobile devices, data centers, and even supercomputers. The number of parameters needed in CNNs, however, are often large and undesirable. Consequently, various methods have been developed to prune a CNN once it is trained. Nevertheless, the resulting CNNs offer limited benefits. While pruning the fully connected layers reduces a CNN's size considerably, it does not improve inference speed noticeably as the compute heavy parts lie in convolutions. Pruning CNNs in a way that increase inference speed often imposes specific sparsity structures, thus limiting the achievable sparsity levels. We present a method to realize simultaneously size economy and speed improvement while pruning CNNs. Paramount to our success is an efficient general sparse-with-dense matrix multiplication implementation that is applicable to convolution of feature maps with kernels of arbitrary sparsity patterns. Complementing this, we developed a performance model that predicts sweet spots of sparsity levels for different layers and on different computer architectures. Together, these two allow us to demonstrate 3.1--7.3$\times$ convolution speedups over dense convolution in AlexNet, on Intel Atom, Xeon, and Xeon Phi processors, spanning the spectrum from mobile devices to supercomputers. We also open source our project at https://github.com/IntelLabs/SkimCaffe.

cs.CV↗

arXiv2017

Coordinating Filters for Faster Deep Neural Networks

Very large-scale Deep Neural Networks (DNNs) have achieved remarkable successes in a large variety of computer vision tasks. However, the high computation intensity of DNNs makes it challenging to deploy these models on resource-limited systems. Some studies used low-rank approaches that approximate the filters by low-rank basis to accelerate the testing. Those works directly decomposed the pre-trained DNNs by Low-Rank Approximations (LRA). How to train DNNs toward lower-rank space for more efficient DNNs, however, remains as an open area. To solve the issue, in this work, we propose Force Regularization, which uses attractive forces to enforce filters so as to coordinate more weight information into lower-rank space. We mathematically and empirically verify that after applying our technique, standard LRA methods can reconstruct filters using much lower basis and thus result in faster DNNs. The effectiveness of our approach is comprehensively evaluated in ResNets, AlexNet, and GoogLeNet. In AlexNet, for example, Force Regularization gains 2x speedup on modern GPU without accuracy loss and 4.05x speedup on CPU by paying small accuracy degradation. Moreover, Force Regularization better initializes the low-rank DNNs such that the fine-tuning can converge faster toward higher accuracy. The obtained lower-rank DNNs can be further sparsified, proving that Force Regularization can be integrated with state-of-the-art sparsity-based acceleration methods. Source code is available in https://github.com/wenwei202/caffe

cs.CV↗

arXiv2017

Group Scissor: Scaling Neuromorphic Computing Design to Large Neural Networks

Synapse crossbar is an elementary structure in Neuromorphic Computing Systems (NCS). However, the limited size of crossbars and heavy routing congestion impedes the NCS implementations of big neural networks. In this paper, we propose a two-step framework (namely, group scissor) to scale NCS designs to big neural networks. The first step is rank clipping, which integrates low-rank approximation into the training to reduce total crossbar area. The second step is group connection deletion, which structurally prunes connections to reduce routing congestion between crossbars. Tested on convolutional neural networks of LeNet on MNIST database and ConvNet on CIFAR-10 database, our experiments show significant reduction of crossbar area and routing area in NCS designs. Without accuracy loss, rank clipping reduces total crossbar area to 13.62\% and 51.81\% in the NCS designs of LeNet and ConvNet, respectively. Following rank clipping, group connection deletion further reduces the routing area of LeNet and ConvNet to 8.1\% and 52.06\%, respectively.

cs.NE↗

arXiv2017

A Compact DNN: Approaching GoogLeNet-Level Accuracy of Classification and Domain Adaptation

Recently, DNN model compression based on network architecture design, e.g., SqueezeNet, attracted a lot attention. No accuracy drop on image classification is observed on these extremely compact networks, compared to well-known models. An emerging question, however, is whether these model compression techniques hurt DNN's learning ability other than classifying images on a single dataset. Our preliminary experiment shows that these compression methods could degrade domain adaptation (DA) ability, though the classification performance is preserved. Therefore, we propose a new compact network architecture and unsupervised DA method in this paper. The DNN is built on a new basic module Conv-M which provides more diverse feature extractors without significantly increasing parameters. The unified framework of our DA method will simultaneously learn invariance across domains, reduce divergence of feature representations, and adapt label prediction. Our DNN has 4.1M parameters, which is only 6.7% of AlexNet or 59% of GoogLeNet. Experiments show that our DNN obtains GoogLeNet-level accuracy both on classification and DA, and our DA method slightly outperforms previous competitive ones. Put all together, our DA strategy based on our DNN achieves state-of-the-art on sixteen of total eighteen DA tasks on popular Office-31 and Office-Caltech datasets.

cs.CV↗

arXiv2017

Generative Poisoning Attack Method Against Neural Networks

Poisoning attack is identified as a severe security threat to machine learning algorithms. In many applications, for example, deep neural network (DNN) models collect public data as the inputs to perform re-training, where the input data can be poisoned. Although poisoning attack against support vector machines (SVM) has been extensively studied before, there is still very limited knowledge about how such attack can be implemented on neural networks (NN), especially DNNs. In this work, we first examine the possibility of applying traditional gradient-based method (named as the direct gradient method) to generate poisoned data against NNs by leveraging the gradient of the target model w.r.t. the normal data. We then propose a generative method to accelerate the generation rate of the poisoned data: an auto-encoder (generator) used to generate poisoned data is updated by a reward function of the loss, and the target NN model (discriminator) receives the poisoned data to calculate the loss w.r.t. the normal data. Our experiment results show that the generative method can speed up the poisoned data generation rate by up to 239.38x compared with the direct gradient method, with slightly lower model accuracy degradation. A countermeasure is also designed to detect such poisoning attack methods by checking the loss of the target model.

cs.CR↗

arXiv2017

Classification Accuracy Improvement for Neuromorphic Computing Systems with One-level Precision Synapses

Brain inspired neuromorphic computing has demonstrated remarkable advantages over traditional von Neumann architecture for its high energy efficiency and parallel data processing. However, the limited resolution of synaptic weights degrades system accuracy and thus impedes the use of neuromorphic systems. In this work, we propose three orthogonal methods to learn synapses with one-level precision, namely, distribution-aware quantization, quantization regularization and bias tuning, to make image classification accuracy comparable to the state-of-the-art. Experiments on both multi-layer perception and convolutional neural networks show that the accuracy drop can be well controlled within 0.19% (5.53%) for MNIST (CIFAR-10) database, compared to an ideal system without quantization.

cs.NE↗

arXiv2016

Learning Structured Sparsity in Deep Neural Networks

High demand for computation resources severely hinders deployment of large-scale Deep Neural Networks (DNN) in resource constrained devices. In this work, we propose a Structured Sparsity Learning (SSL) method to regularize the structures (i.e., filters, channels, filter shapes, and layer depth) of DNNs. SSL can: (1) learn a compact structure from a bigger DNN to reduce computation cost; (2) obtain a hardware-friendly structured sparsity of DNN to efficiently accelerate the DNNs evaluation. Experimental results show that SSL achieves on average 5.1x and 3.1x speedups of convolutional layer computation of AlexNet against CPU and GPU, respectively, with off-the-shelf libraries. These speedups are about twice speedups of non-structured sparsity; (3) regularize the DNN structure to improve classification accuracy. The results show that for CIFAR-10, regularization on layer depth can reduce 20 layers of a Deep Residual Network (ResNet) to 18 layers while improve the accuracy from 91.25% to 92.60%, which is still slightly higher than that of original ResNet with 32 layers. For AlexNet, structure regularization by SSL also reduces the error by around ~1%. Open source code is in https://github.com/wenwei202/caffe/tree/scnn

cs.NE↗

arXiv2016

A New Learning Method for Inference Accuracy, Core Occupation, and Performance Co-optimization on TrueNorth Chip

IBM TrueNorth chip uses digital spikes to perform neuromorphic computing and achieves ultrahigh execution parallelism and power efficiency. However, in TrueNorth chip, low quantization resolution of the synaptic weights and spikes significantly limits the inference (e.g., classification) accuracy of the deployed neural network model. Existing workaround, i.e., averaging the results over multiple copies instantiated in spatial and temporal domains, rapidly exhausts the hardware resources and slows down the computation. In this work, we propose a novel learning method on TrueNorth platform that constrains the random variance of each computation copy and reduces the number of needed copies. Compared to the existing learning method, our method can achieve up to 68.8% reduction of the required neuro-synaptic cores or 6.5X speedup, with even slightly improved inference accuracy.

cs.NE↗

arXiv2014

A Universal, Rapid Method for Clean Transfer of Nanostructures onto Various Substrates

Transfer and integration of nanostructures onto target substrates is the prerequisite for their fundamental studies and practical applications. Conventional transfer techniques that involve stamping, lift-off and/or striping suffer from the process-specific drawbacks, such as the requirement for chemical etchant or high-temperature annealing and the introduction of surface discontinuities and/or contaminations that can greatly hinder the properties and functions of the transferred materials. Herein, we report a universal and rapid transfer method implementable at mild conditions. Nanostructures with various dimensionalities (i.e. nanoparticles, nanowires and nanosheets) and surface properties (i.e. hydrophilic and hydrophobic) can be facilely transferred to diverse substrates including hydrophilic, hydrophobic and flexible surfaces with good fidelity. Importantly, our method ensures the rapid and clean transfer of two-dimensional (2D) materials, and allows for the facile fabrication of vertical heterostructures with various compositions used for electronic devices. We believe that our method can facilitate the development of nano-electronics by accelerating the clean transfer and integration of low-dimensional materials into multidimensional structures.

cond-mat.mtrl-sci↗

arXiv2014

Quantum coherence rather than quantum correlations reflect the effects of reservoir on the system's work capability

We consider a model of an optical cavity with a nonequilibrium reservoir consisting of a beam of identical two-level atom pairs (TLAPs) in the general X-state. We find that coherence of multiparticle nonequilibrium reservoir plays a central role on the potential work capability of cavity. We show that no matter whether there are quantum correlations in each TLAP (including quantum entanglement and quantum discord) or not the coherence of the TLAPs has an effect on the work capability of the cavity. Additionally, constructive and destructive interferences could be induced to influence the work capability of cavity only by adjusting the relative phase with which quantum correlations have nothing to do. In this paper, the coherence of reservoir rather than the quantum correlations effectively reflecting the effects of reservoir on the system's work capability is demonstrated clearly.

quant-ph↗

arXiv2013

Single-Layer MoS2 Phototransistors

A new phototransistor based on the mechanically-exfoliated single-layer MoS2 nanosheet is fabricated and its light-induced electric properties are investigated in details. Photocurrent generated from the phototransistor is solely determined by the illuminated optical power at a constant drain or gate voltage. The switching behavior of photocurrent generation and annihilation can be completely finished within ca. 50 ms and it shows good stability. Especially, the single-layer MoS2 phototransistor exhibits a better photoresponsivity as compared with the graphene-based device. The unique characteristics of incident-light control, prompt photoswitching and good photoresponsivity from the MoS2 phototransistor pave an avenue to develop the single-layer semiconducting materials for multi-functional optoelectronic device applications in future.

cond-mat.mtrl-sci↗

arXiv2013

Investigation of MoS2 and Graphene Nanosheets by Magnetic Force Microscopy

For the first time, the magnetic force microscopy (MFM) is used to characterize the mechanically-exfoliated single- and few-layer MoS2 and graphene nanosheets. By analysis of the phase and amplitude shifts, the magnetic response of MoS2 and graphene nanosheets exhibits the dependence on their layer number. However, the solution-processed single-layer MoS2 nanosheet shows the reverse magnetic signal to the mechanically-exfoliated one, and the graphene oxide nanosheet has not shown any detectable magnetic signal. Importantly, graphene and MoS2 flakes become nonmagnetic when they exceed a certain thickness.

cond-mat.mtrl-sci↗

arXiv2013

Rapid and reliable thickness identification of two-dimensional nanosheets using optical microscopy

The physical and electronic properties of ultrathin two-dimensional (2D) layered nanomaterials are highly related to their thickness. Therefore, the rapid and accurate identification of single- and few- to multi-layer nanosheets is essential to their fundamental study and practical applications. Here, a universal optical method has been developed for simple, rapid and reliable identification of single- to quindecuple-layer (1L-15L) 2D nanosheets, including graphene, MoS2, WSe2 and TaS2, on Si substrates coated with 90 nm or 300 nm SiO2. The optical contrast differences between the substrates and 2D nanosheets with different layer numbers were collected and tabulated, serving as a standard reference, from which the layer number of a given nanosheet can be readily and reliably determined without using complex calculation nor expensive instrument. Our general optical identification method will facilitate the thickness-dependent study of various 2D nanomaterials, and expedite their research toward practical applications.

cond-mat.mtrl-sci↗

arXiv2013

Interlayer breathing and shear modes in few-trilayer MoS2 and WSe2

Two-dimensional (2D) layered transition metal dichalcogenides (TMDs) have recently attracted tremendous interest as potential valleytronic and nano-electronic materials, in addition to being well-known as excellent lubricants in the bulk. The interlayer van der Waals (vdW) coupling and low frequency phonon modes, and how they evolve with the number of layers, are important for both the mechanical and electrical properties of 2D TMDs. Here we uncover the ultra-low frequency interlayer breathing and shear modes in few-layer MoS2 and WSe2, prototypical layered TMDs, using both Raman spectroscopy and first principles calculations. Remarkably, the frequencies of these modes can be perfectly described using a simple linear chain model with only nearest-neighbour interactions. We show that the derived in-plane (shear) and out-of-plane (breathing) force constants from experiment remain the same from two-layer 2D crystals to the bulk materials, suggesting that the nanoscale interlayer frictional characteristics of these excellent lubricants should be independent of the number of layers.

cond-mat.mtrl-sci↗