arXiv subjects

Zhe Li

Publications and source records attributed to Zhe Li.

At least 289 records · Page 16Linked to original sources

arXiv2019

EIGEN: Ecologically-Inspired GENetic Approach for Neural Network Structure Searching from Scratch

Designing the structure of neural networks is considered one of the most challenging tasks in deep learning, especially when there is few prior knowledge about the task domain. In this paper, we propose an Ecologically-Inspired GENetic (EIGEN) approach that uses the concept of succession, extinction, mimicry, and gene duplication to search neural network structure from scratch with poorly initialized simple network and few constraints forced during the evolution, as we assume no prior knowledge about the task domain. Specifically, we first use primary succession to rapidly evolve a population of poorly initialized neural network structures into a more diverse population, followed by a secondary succession stage for fine-grained searching based on the networks from the primary succession. Extinction is applied in both stages to reduce computational cost. Mimicry is employed during the entire evolution process to help the inferior networks imitate the behavior of a superior network and gene duplication is utilized to duplicate the learned blocks of novel structures, both of which help to find better network structures. Experimental results show that our proposed approach can achieve similar or better performance compared to the existing genetic approaches with dramatically reduced computation cost. For example, the network discovered by our approach on CIFAR-100 dataset achieves 78.1% test accuracy under 120 GPU hours, compared to 77.0% test accuracy in more than 65, 536 GPU hours in [35].

cs.NE↗

arXiv2019

Doppler Fluctuation Spectroscopy of Intracellular Dynamics in Living Tissue

Intracellular dynamics in living tissue are dominated by active transport driven by bioenergetic processes far from thermal equilibrium. Intracellular constituents typically execute persistent walks. In the limit of long mean-free paths, the persistent walks are ballistic, exhibiting a "Doppler edge" in light scattering fluctuation spectra. At shorter transport lengths, the fluctuations are described by lifetime-broadened Doppler spectra. Dynamic light scattering from transport in the ballistic, diffusive or the cross-over regime is derived analytically, including the derivation of autocorrelation functions through a driven damped harmonic oscillator analog for light scattering from persistent walks. The theory is validated through Monte Carlo simulations. Experimental evidence for the Doppler edge in 3D living tissue is obtained using biodynamic imaging based on low-coherence interferometry and digital holography.

q-bio.QM↗

arXiv2019

Generative Adversarial Active Learning for Unsupervised Outlier Detection

Outlier detection is an important topic in machine learning and has been used in a wide range of applications. In this paper, we approach outlier detection as a binary-classification issue by sampling potential outliers from a uniform reference distribution. However, due to the sparsity of data in high-dimensional space, a limited number of potential outliers may fail to provide sufficient information to assist the classifier in describing a boundary that can separate outliers from normal data effectively. To address this, we propose a novel Single-Objective Generative Adversarial Active Learning (SO-GAAL) method for outlier detection, which can directly generate informative potential outliers based on the mini-max game between a generator and a discriminator. Moreover, to prevent the generator from falling into the mode collapsing problem, the stop node of training should be determined when SO-GAAL is able to provide sufficient information. But without any prior information, it is extremely difficult for SO-GAAL. Therefore, we expand the network structure of SO-GAAL from a single generator to multiple generators with different objectives (MO-GAAL), which can generate a reasonable reference distribution for the whole dataset. We empirically compare the proposed approach with several state-of-the-art outlier detection methods on both synthetic and real-world datasets. The results show that MO-GAAL outperforms its competitors in the majority of cases, especially for datasets with various cluster types or high irrelevant variable ratio.

cs.LG↗

arXiv2019

CircConv: A Structured Convolution with Low Complexity

Deep neural networks (DNNs), especially deep convolutional neural networks (CNNs), have emerged as the powerful technique in various machine learning applications. However, the large model sizes of DNNs yield high demands on computation resource and weight storage, thereby limiting the practical deployment of DNNs. To overcome these limitations, this paper proposes to impose the circulant structure to the construction of convolutional layers, and hence leads to circulant convolutional layers (CircConvs) and circulant CNNs. The circulant structure and models can be either trained from scratch or re-trained from a pre-trained non-circulant model, thereby making it very flexible for different training environments. Through extensive experiments, such strong structure-imposing approach is proved to be able to substantially reduce the number of parameters of convolutional layers and enable significant saving of computational cost by using fast multiplication of the circulant tensor.

cs.CV↗

arXiv2018

E-RNN: Design Optimization for Efficient Recurrent Neural Networks in FPGAs

Recurrent Neural Networks (RNNs) are becoming increasingly important for time series-related applications which require efficient and real-time implementations. The two major types are Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) networks. It is a challenging task to have real-time, efficient, and accurate hardware RNN implementations because of the high sensitivity to imprecision accumulation and the requirement of special activation function implementations. A key limitation of the prior works is the lack of a systematic design optimization framework of RNN model and hardware implementations, especially when the block size (or compression ratio) should be jointly optimized with RNN type, layer size, etc. In this paper, we adopt the block-circulant matrix-based framework, and present the Efficient RNN (E-RNN) framework for FPGA implementations of the Automatic Speech Recognition (ASR) application. The overall goal is to improve performance/energy efficiency under accuracy requirement. We use the alternating direction method of multipliers (ADMM) technique for more accurate block-circulant training, and present two design explorations providing guidance on block size and reducing RNN training trials. Based on the two observations, we decompose E-RNN in two phases: Phase I on determining RNN model to reduce computation and storage subject to accuracy requirement, and Phase II on hardware implementations given RNN model, including processing element design/optimization, quantization, activation implementation, etc. Experimental results on actual FPGA deployments show that E-RNN achieves a maximum energy efficiency improvement of 37.4$\times$ compared with ESE, and more than 2$\times$ compared with C-LSTM, under the same accuracy.

cs.CV↗

arXiv2018

Realizing an Epitaxial Stanene with an Insulating Bandgap

Stanene, a single atomic layer of Sn in a honeycomb lattice, is predicted a candidate wide bandgap two-dimensional (2D) topological insulator and can host intriguing topological states of matter such as quantum anomalous Hall effect and topological superconductivity with different surface modifications. Despite intensive research efforts, one still cannot obtain bulk-insulating stanene samples-a prerequisite for any transport studies and applications of stanene. Here we show the experimental realization of an epitaxial stanene with an insulating bulk bandgap by using PbTe(111) substrates. With low-temperature molecular beam epitaxy, we are able to grow single layer stanene on PbTe(111). In-situ angle-resolved photoemission spectroscopy shows the characteristic stanene bands with its Fermi level lying in the bandgap. Doping Sr in PbTe removes the substrate states located in the stanene gap, resulting in a stanene sample with truly insulating bulk. This experimental progress paves the way for studies of stanene-based topological quantum effects.

cond-mat.mtrl-sci↗

arXiv2018

A Unified Analysis of Stochastic Momentum Methods for Deep Learning

Stochastic momentum methods have been widely adopted in training deep neural networks. However, their theoretical analysis of convergence of the training objective and the generalization error for prediction is still under-explored. This paper aims to bridge the gap between practice and theory by analyzing the stochastic gradient (SG) method, and the stochastic momentum methods including two famous variants, i.e., the stochastic heavy-ball (SHB) method and the stochastic variant of Nesterov's accelerated gradient (SNAG) method. We propose a framework that unifies the three variants. We then derive the convergence rates of the norm of gradient for the non-convex optimization problem, and analyze the generalization performance through the uniform stability approach. Particularly, the convergence analysis of the training objective exhibits that SHB and SNAG have no advantage over SG. However, the stability analysis shows that the momentum term can improve the stability of the learned model and hence improve the generalization performance. These theoretical insights verify the common wisdom and are also corroborated by our empirical analysis on deep learning.

cs.LG↗

arXiv2018

C3PO: Database and Benchmark for Early-stage Malicious Activity Detection in 3D Printing

Increasing malicious users have sought practices to leverage 3D printing technology to produce unlawful tools in criminal activities. Current regulations are inadequate to deal with the rapid growth of 3D printers. It is of vital importance to enable 3D printers to identify the objects to be printed, so that the manufacturing procedure of an illegal weapon can be terminated at the early stage. Deep learning yields significant rises in performance in the object recognition tasks. However, the lack of large-scale databases in 3D printing domain stalls the advancement of automatic illegal weapon recognition. This paper presents a new 3D printing image database, namely C3PO, which compromises two subsets for the different system working scenarios. We extract images from the numerical control programming code files of 22 3D models, and then categorize the images into 10 distinct labels. The first set consists of 62,200 images which represent the object projections on the three planes in a Cartesian coordinate system. And the second sets consists of sequences of total 671,677 images to simulate the cameras' captures of the printed objects. Importantly, we demonstrate that the weapons can be recognized in either scenario using deep learning based approaches using our proposed database. % We also use the trained deep models to build a prototype of object-aware 3D printer. The quantitative results are promising, and the future exploration of the database and the crime prevention in 3D printing are demanding tasks.

cs.CV↗

arXiv2018

An Augmented Nonlinear LMS for Digital Self-Interference Cancellation in Full-Duplex Direct-Conversion Transceivers

In future full-duplex communications, the cancellation of self-interference (SI) arising from hardware non-idealities will play an important role in the design of mobile-scale devices. To this end, we introduce an optimal digital SI cancellation solution for shared-antenna-based direct-conversion transceivers. To establish that the underlying widely linear signal model is not adequate for strong transmit signals, the impact of various circuit imperfections, including power amplifier (PA) distortion, frequency-dependent I/Q imbalance, quantization noise and thermal noise, on the performance of the conventional augmented least mean square (LMS) based SI canceller, is analyzed. In order to achieve a sufficient signal-to-interference-plus-noise ratio (SINR) when the nonlinear SI components are not negligible, we propose an augmented nonlinear LMS based SI canceller for a joint cancellation of both the linear and nonlinear SI components by virtue of a widely nonlinear model fit. A rigorous mean and mean square performance evaluation is conducted to justify the performance advantages of the proposed scheme over the conventional augmented LMS solution. Simulations on orthogonal frequency division multiplexing (OFDM)-based wireless local area network (WLAN) standard compliant waveforms support the analysis.

cs.IT↗

arXiv2018

An Adaptive Volumetric Flux Boundary Condition for Lattice Boltzmann Methods

This paper presents a spatially and temporally adaptive boundary condition to specify the volumetric flux for lattice Boltzmann methods. The approach differs from standard velocity boundary conditions because it allows the velocity to vary over the boundary region provided that the total flux through the boundary satisfies a prescribed constraint, which is a typical scenario for laboratory experimental studies. This condition allows the boundary pressure to adjust dynamically to yield a specified boundary flow rate as a means to avoid unphysical mismatch between the boundary velocity and the interior flow field that can arise when a constant velocity boundary condition is applied. The method is validated for simulation of one- and two-fluid flow in complex materials, with conditions determined to match typical experiments used to study flow in porous media.

physics.comp-ph↗

arXiv2018

Thoracic Disease Identification and Localization with Limited Supervision

Accurate identification and localization of abnormalities from radiology images play an integral part in clinical diagnosis and treatment planning. Building a highly accurate prediction model for these tasks usually requires a large number of images manually annotated with labels and finding sites of abnormalities. In reality, however, such annotated data are expensive to acquire, especially the ones with location annotations. We need methods that can work well with only a small amount of location annotations. To address this challenge, we present a unified approach that simultaneously performs disease identification and localization through the same underlying model for all images. We demonstrate that our approach can effectively leverage both class information as well as limited location annotation, and significantly outperforms the comparative reference baseline in both classification and localization tasks.

cs.CV↗

arXiv2018

An Aggressive Genetic Programming Approach for Searching Neural Network Structure Under Computational Constraints

Recently, there emerged revived interests of designing automatic programs (e.g., using genetic/evolutionary algorithms) to optimize the structure of Convolutional Neural Networks (CNNs) for a specific task. The challenge in designing such programs lies in how to balance between large search space of the network structures and high computational costs. Existing works either impose strong restrictions on the search space or use enormous computing resources. In this paper, we study how to design a genetic programming approach for optimizing the structure of a CNN for a given task under limited computational resources yet without imposing strong restrictions on the search space. To reduce the computational costs, we propose two general strategies that are observed to be helpful: (i) aggressively selecting strongest individuals for survival and reproduction, and killing weaker individuals at a very early age; (ii) increasing mutation frequency to encourage diversity and faster evolution. The combined strategy with additional optimization techniques allows us to explore a large search space but with affordable computational costs. Our results on standard benchmark datasets (MNIST, SVHN, CIFAR-10, CIFAR-100) are competitive to similar approaches with significantly reduced computational costs.

cs.NE↗

arXiv2018

Towards Budget-Driven Hardware Optimization for Deep Convolutional Neural Networks using Stochastic Computing

Recently, Deep Convolutional Neural Network (DCNN) has achieved tremendous success in many machine learning applications. Nevertheless, the deep structure has brought significant increases in computation complexity. Largescale deep learning systems mainly operate in high-performance server clusters, thus restricting the application extensions to personal or mobile devices. Previous works on GPU and/or FPGA acceleration for DCNNs show increasing speedup, but ignore other constraints, such as area, power, and energy. Stochastic Computing (SC), as a unique data representation and processing technique, has the potential to enable the design of fully parallel and scalable hardware implementations of large-scale deep learning systems. This paper proposed an automatic design allocation algorithm driven by budget requirement considering overall accuracy performance. This systematic method enables the automatic design of a DCNN where all design parameters are jointly optimized. Experimental results demonstrate that proposed algorithm can achieve a joint optimization of all design parameters given the comprehensive budget of a DCNN.

cs.NE↗

arXiv2018

Learning Topics using Semantic Locality

The topic modeling discovers the latent topic probability of the given text documents. To generate the more meaningful topic that better represents the given document, we proposed a new feature extraction technique which can be used in the data preprocessing stage. The method consists of three steps. First, it generates the word/word-pair from every single document. Second, it applies a two-way TF-IDF algorithm to word/word-pair for semantic filtering. Third, it uses the K-means algorithm to merge the word pairs that have the similar semantic meaning. Experiments are carried out on the Open Movie Database (OMDb), Reuters Dataset and 20NewsGroup Dataset. The mean Average Precision score is used as the evaluation metric. Comparing our results with other state-of-the-art topic models, such as Latent Dirichlet allocation and traditional Restricted Boltzmann Machines. Our proposed data preprocessing can improve the generated topic accuracy by up to 12.99\%.

cs.LG↗

arXiv2018

Image Dataset for Visual Objects Classification in 3D Printing

The rapid development in additive manufacturing (AM), also known as 3D printing, has brought about potential risk and security issues along with significant benefits. In order to enhance the security level of the 3D printing process, the present research aims to detect and recognize illegal components using deep learning. In this work, we collected a dataset of 61,340 2D images (28x28 for each image) of 10 classes including guns and other non-gun objects, corresponding to the projection results of the original 3D models. To validate the dataset, we train a convolutional neural network (CNN) model for gun classification which can achieve 98.16% classification accuracy.

cs.CV↗

arXiv2018

Efficient Recurrent Neural Networks using Structured Matrices in FPGAs

Recurrent Neural Networks (RNNs) are becoming increasingly important for time series-related applications which require efficient and real-time implementations. The recent pruning based work ESE suffers from degradation of performance/energy efficiency due to the irregular network structure after pruning. We propose block-circulant matrices for weight matrix representation in RNNs, thereby achieving simultaneous model compression and acceleration. We aim to implement RNNs in FPGA with highest performance and energy efficiency, with certain accuracy requirement (negligible accuracy degradation). Experimental results on actual FPGA deployments shows that the proposed framework achieves a maximum energy efficiency improvement of 35.7$\times$ compared with ESE.

cs.LG↗

arXiv2018

C-LSTM: Enabling Efficient LSTM using Structured Compression Techniques on FPGAs

Recently, significant accuracy improvement has been achieved for acoustic recognition systems by increasing the model size of Long Short-Term Memory (LSTM) networks. Unfortunately, the ever-increasing size of LSTM model leads to inefficient designs on FPGAs due to the limited on-chip resources. The previous work proposes to use a pruning based compression technique to reduce the model size and thus speedups the inference on FPGAs. However, the random nature of the pruning technique transforms the dense matrices of the model to highly unstructured sparse ones, which leads to unbalanced computation and irregular memory accesses and thus hurts the overall performance and energy efficiency. In contrast, we propose to use a structured compression technique which could not only reduce the LSTM model size but also eliminate the irregularities of computation and memory accesses. This approach employs block-circulant instead of sparse matrices to compress weight matrices and reduces the storage requirement from $\mathcal{O}(k^2)$ to $\mathcal{O}(k)$. Fast Fourier Transform algorithm is utilized to further accelerate the inference by reducing the computational complexity from $\mathcal{O}(k^2)$ to $\mathcal{O}(k\text{log}k)$. The datapath and activation functions are quantized as 16-bit to improve the resource utilization. More importantly, we propose a comprehensive framework called C-LSTM to automatically optimize and implement a wide range of LSTM variants on FPGAs. According to the experimental results, C-LSTM achieves up to 18.8X and 33.5X gains for performance and energy efficiency compared with the state-of-the-art LSTM implementation under the same experimental setup, and the accuracy degradation is very small.

cs.LG↗

arXiv2018

A Framework in CRM Customer Lifecycle: Identify Downward Trend and Potential Issues Detection

Customer retention is one of the primary goals in the area of customer relationship management. A mass of work exists in which machine learning models or business rules are established to predict churn. However, targeting users at an early stage when they start to show a downward trend is a better strategy. In downward trend prediction, the reasons why customers show a downward trend is of great interest in the industry as it helps the business to understand the pain points that customers suffer and to take early action to prevent them from churning. A commonly used method is to collect feedback from customers by either aggressively reaching out to them or by passively hearing from them. However, it is believed that there are a large number of customers who have unpleasant experiences and never speak out. In the literature, there is limited research work that provides a comprehensive and scientific approach to identify these "silent suffers". In this study, we propose a novel two-part framework: developing the downward prediction process and establishing the methodology to identify the reasons why customers are in the downward trend. In the first prediction part, we focus on predicting the downward trend, which is an earlier stage of the customer lifecycle compared to churn. In the second part, we propose an approach to figuring out the cause (of the downward trend) based on a causal inference method and semi-supervised learning. The proposed approach is capable of identifying potential silent sufferers. We take bad shopping experiences as inputs to develop the framework and validate it via a marketing A/B test in the real world. The test readout demonstrates the effectiveness of the framework by driving 88.5% incremental lift in purchase volume.

cs.CY↗