arXiv subjects

Jack Xin

Publications and source records attributed to Jack Xin.

At least 91 records · Page 5Linked to original sources

arXiv2019

Convergence of a Relaxed Variable Splitting Coarse Gradient Descent Method for Learning Sparse Weight Binarized Activation Neural Networks

Sparsification of neural networks is one of the effective complexity reduction methods to improve efficiency and generalizability. Binarized activation offers an additional computational saving for inference. Due to vanishing gradient issue in training networks with binarized activation, coarse gradient (a.k.a. straight through estimator) is adopted in practice. In this paper, we study the problem of coarse gradient descent (CGD) learning of a one hidden layer convolutional neural network (CNN) with binarized activation function and sparse weights. It is known that when the input data is Gaussian distributed, no-overlap one hidden layer CNN with ReLU activation and general weight can be learned by GD in polynomial time at high probability in regression problems with ground truth. We propose a relaxed variable splitting method integrating thresholding and coarse gradient descent. The sparsity in network weight is realized through thresholding during the CGD training process. We prove that under threshholding of $\ell_1, \ell_0,$ and transformed-$\ell_1$ penalties, no-overlap binary activation CNN can be learned with high probability, and the iterative weights converge to a global limit which is a transformation of the true weight under a novel sparsifying operation. We found explicit error estimates of sparse weights from the true weights.

math.OC↗

arXiv2019

AutoShuffleNet: Learning Permutation Matrices via an Exact Lipschitz Continuous Penalty in Deep Convolutional Neural Networks

ShuffleNet is a state-of-the-art light weight convolutional neural network architecture. Its basic operations include group, channel-wise convolution and channel shuffling. However, channel shuffling is manually designed empirically. Mathematically, shuffling is a multiplication by a permutation matrix. In this paper, we propose to automate channel shuffling by learning permutation matrices in network training. We introduce an exact Lipschitz continuous non-convex penalty so that it can be incorporated in the stochastic gradient descent to approximate permutation at high precision. Exact permutations are obtained by simple rounding at the end of training and are used in inference. The resulting network, referred to as AutoShuffleNet, achieved improved classification accuracies on CIFAR-10 and ImageNet data sets. In addition, we found experimentally that the standard convex relaxation of permutation matrices into stochastic matrices leads to poor performance. We prove theoretically the exactness (error bounds) in recovering permutation matrices when our penalty function is zero (very small). We present examples of permutation optimization through graph matching and two-layer neural network models where the loss functions are calculated in closed analytical form. In the examples, convex relaxation failed to capture permutations whereas our penalty succeeded.

cs.LG↗

arXiv2019

Blended Coarse Gradient Descent for Full Quantization of Deep Neural Networks

Quantized deep neural networks (QDNNs) are attractive due to their much lower memory storage and faster inference speed than their regular full precision counterparts. To maintain the same performance level especially at low bit-widths, QDNNs must be retrained. Their training involves piecewise constant activation functions and discrete weights, hence mathematical challenges arise. We introduce the notion of coarse gradient and propose the blended coarse gradient descent (BCGD) algorithm, for training fully quantized neural networks. Coarse gradient is generally not a gradient of any function but an artificial ascent direction. The weight update of BCGD goes by coarse gradient correction of a weighted average of the full precision weights and their quantization (the so-called blending), which yields sufficient descent in the objective value and thus accelerates the training. Our experiments demonstrate that this simple blending technique is very effective for quantization at extremely low bit-width such as binarization. In full quantization of ResNet-18 for ImageNet classification task, BCGD gives 64.36\% top-1 accuracy with binary weights across all layers and 4-bit adaptive activation. If the weights in the first and last layers are kept in full precision, this number increases to 65.46\%. As theoretical justification, we show convergence analysis of coarse gradient descent for a two-linear-layer neural network model with Gaussian input data, and prove that the expected coarse gradient correlates positively with the underlying true gradient.

cs.LG↗

arXiv2018

BinaryRelax: A Relaxation Approach For Training Deep Neural Networks With Quantized Weights

We propose BinaryRelax, a simple two-phase algorithm, for training deep neural networks with quantized weights. The set constraint that characterizes the quantization of weights is not imposed until the late stage of training, and a sequence of \emph{pseudo} quantized weights is maintained. Specifically, we relax the hard constraint into a continuous regularizer via Moreau envelope, which turns out to be the squared Euclidean distance to the set of quantized weights. The pseudo quantized weights are obtained by linearly interpolating between the float weights and their quantizations. A continuation strategy is adopted to push the weights towards the quantized state by gradually increasing the regularization parameter. In the second phase, exact quantization scheme with a small learning rate is invoked to guarantee fully quantized weights. We test BinaryRelax on the benchmark CIFAR and ImageNet color image datasets to demonstrate the superiority of the relaxed quantization approach and the improved accuracy over the state-of-the-art training methods. Finally, we prove the convergence of BinaryRelax under an approximate orthogonality condition.

cs.CV↗

arXiv2018

Minimization of Transformed $L_1$ Penalty: Theory, Difference of Convex Function Algorithm, and Robust Application in Compressed Sensing

We study the minimization problem of a non-convex sparsity promoting penalty function, the transformed $l_1$ (TL1), and its application in compressed sensing (CS). The TL1 penalty interpolates $l_0$ and $l_1$ norms through a nonnegative parameter $a \in (0,+\infty)$, similar to $l_p$ with $p \in (0,1]$, and is known to satisfy unbiasedness, sparsity and Lipschitz continuity properties. We first consider the constrained minimization problem and discuss the exact recovery of $l_0$ norm minimal solution based on the null space property (NSP). We then prove the stable recovery of $l_0$ norm minimal solution if the sensing matrix $A$ satisfies a restricted isometry property (RIP). Next, we present difference of convex algorithms for TL1 (DCATL1) in computing TL1-regularized constrained and unconstrained problems in CS. The inner loop concerns an $l_1$ minimization problem on which we employ the Alternating Direction Method of Multipliers (ADMM). For the unconstrained problem, we prove convergence of DCATL1 to a stationary point satisfying the first order optimality condition. In numerical experiments, we identify the optimal value $a=1$, and compare DCATL1 with other CS algorithms on two classes of sensing matrices: Gaussian random matrices and over-sampled discrete cosine transform matrices (DCT). We find that for both classes of sensing matrices, the performance of DCATL1 algorithm (initiated with $l_1$ minimization) always ranks near the top (if not the top), and is the most robust choice insensitive to the conditioning of the sensing matrix $A$. DCATL1 is also competitive in comparison with DCA on other non-convex penalty functions commonly used in statistics with two hyperparameters.

cs.IT↗

arXiv2017

Computing effective diffusivity of chaotic and stochastic flows using structure preserving schemes

In this paper we study the problem of computing the effective diffusivity for a particle moving in chaotic and stochastic flows. In addition we numerically investigate the residual diffusion phenomenon in chaotic advection. The residual diffusion refers to the non-zero effective (homogenized) diffusion in the limit of zero molecular diffusion as a result of chaotic mixing of the streamlines. In this limit traditional numerical methods typically fail since the solutions of the advection-diffusion equation develop sharp gradients. Instead of solving the Fokker-Planck equation in the Eulerian formulation, we compute the motion of particles in the Lagrangian formulation, which is modelled by stochastic differential equations (SDEs). We propose a new numerical integrator based on a stochastic splitting method to solve the corresponding SDEs in which the deterministic subproblem is symplectic preserving while the random subproblem can be viewed as a perturbation. We provide rigorous error analysis for the new numerical integrator using the backward error analysis technique and show that our method outperforms standard Euler-based integrators. Numerical results are presented to demonstrate the accuracy and efficiency of the proposed method for several typical chaotic and stochastic flow problems of physical interests.

math.NA↗

arXiv2017

Deep Learning for Real-Time Crime Forecasting and its Ternarization

Real-time crime forecasting is important. However, accurate prediction of when and where the next crime will happen is difficult. No known physical model provides a reasonable approximation to such a complex system. Historical crime data are sparse in both space and time and the signal of interests is weak. In this work, we first present a proper representation of crime data. We then adapt the spatial temporal residual network on the well represented data to predict the distribution of crime in Los Angeles at the scale of hours in neighborhood-sized parcels. These experiments as well as comparisons with several existing approaches to prediction demonstrate the superiority of the proposed model in terms of accuracy. Finally, we present a ternarization technique to address the resource consumption issue for its deployment in real world. This work is an extension of our short conference proceeding paper [Wang et al, Arxiv 1707.03340].

cs.LG↗

arXiv2017

Quantization and Training of Low Bit-Width Convolutional Neural Networks for Object Detection

We present LBW-Net, an efficient optimization based method for quantization and training of the low bit-width convolutional neural networks (CNNs). Specifically, we quantize the weights to zero or powers of two by minimizing the Euclidean distance between full-precision weights and quantized weights during backpropagation. We characterize the combinatorial nature of the low bit-width quantization problem. For 2-bit (ternary) CNNs, the quantization of $N$ weights can be done by an exact formula in $O(N\log N)$ complexity. When the bit-width is three and above, we further propose a semi-analytical thresholding scheme with a single free parameter for quantization that is computationally inexpensive. The free parameter is further determined by network retraining and object detection tests. LBW-Net has several desirable advantages over full-precision CNNs, including considerable memory savings, energy efficiency, and faster deployment. Our experiments on PASCAL VOC dataset show that compared with its 32-bit floating-point counterpart, the performance of the 6-bit LBW-Net is nearly lossless in the object detection tasks, and can even do better in some real world visual scenes, while empirically enjoying more than 4$\times$ faster deployment.

cs.LG↗

arXiv2017

$\ell_1$-minimization method for link flow correction

A computational method, based on $\ell_1$-minimization, is proposed for the problem of link flow correction, when the available traffic flow data on many links in a road network are inconsistent with respect to the flow conservation law. Without extra information, the problem is generally ill-posed when a large portion of the link sensors are unhealthy. It is possible, however, to correct the corrupted link flows \textit{accurately} with the proposed method under a recoverability condition if there are only a few bad sensors which are located at certain links. We analytically identify the links that are robust to miscounts and relate them to the geometric structure of the traffic network by introducing the recoverability concept and an algorithm for computing it. The recoverability condition for corrupted links is simply the associated recoverability being greater than 1. In a more realistic setting, besides the unhealthy link sensors, small measurement noises may be present at the other sensors. Under the same recoverability condition, our method guarantees to give an estimated traffic flow fairly close to the ground-truth data and leads to a bound for the correction error. Both synthetic and real-world examples are provided to demonstrate the effectiveness of the proposed method.

cs.IT↗

arXiv2017

Residual Diffusivity in Elephant Random Walk Models with Stops

We study the enhanced diffusivity in the so called elephant random walk model with stops (ERWS) by including symmetric random walk steps at small probability $ε$. At any $ε> 0$, the large time behavior transitions from sub-diffusive at $ε= 0$ to diffusive in a wedge shaped parameter regime where the diffusivity is strictly above that in the un-perturbed ERWS model in the $ε\downarrow 0$ limit. The perturbed ERWS model is shown to be solvable with the first two moments and their asymptotics calculated exactly in both one and two space dimensions. The model provides a discrete analytical setting of the residual diffusion phenomenon known for the passive scalar transport in chaotic flows (e.g. generated by time periodic cellular flows and statistically sub-diffusive) as molecular diffusivity tends to zero.

math.PR↗

arXiv2017

Curvature effect in shear flow: slowdown of turbulent flame speeds with Markstein number

It is well known in the combustion community that curvature effect in general slows down flame propagation speeds because it smooths out wrinkled flames. However, such a folklore has never been justified rigorously. In this paper, as the first theoretical result in this direction, we prove that the turbulent flame speed (an effective burning velocity) is decreasing with respect to the curvature diffusivity (Markstein number) for shear flows in the well known G-equation model. Our proof involves several novel and rather sophisticated inequalities arising from the nonlinear structure of the equation. On a related fundamental issue, we solve the selection problem of weak solutions or find the "physical fluctuations" when the Markstein number goes to zero and solutions approach those of the inviscid G-equation model. The limiting solution is given by a closed form analytical formula.

math.AP↗

arXiv2016

Iterative $\ell_1$ minimization for non-convex compressed sensing

An algorithmic framework, based on the difference of convex functions algorithm (DCA), is proposed for minimizing a class of concave sparse metrics for compressed sensing problems. The resulting algorithm iterates a sequence of $\ell_1$ minimization problems. An exact sparse recovery theory is established to show that the proposed framework always improves on the basis pursuit ($\ell_1$ minimization) and inherits robustness from it. Numerical examples on success rates of sparse solution recovery illustrate further that, unlike most existing non-convex compressed sensing solvers in the literature, our method always out-performs basis pursuit, no matter how ill-conditioned the measurement matrix is. Moreover, the iterative $\ell_1$ (IL$_1$) algorithm lead by a wide margin the state-of-the-art algorithms on $\ell_{1/2}$ and logarithimic minimizations in the strongly coherent (highly ill-conditioned) regime, despite the same objective functions. Last but not least, in the application of magnetic resonance imaging (MRI), IL$_1$ algorithm easily recovers the phantom image with just 7 line projections.

cs.IT↗

arXiv2016

Minimization of Transformed $L_1$ Penalty: Closed Form Representation and Iterative Thresholding Algorithms

The transformed $l_1$ penalty (TL1) functions are a one parameter family of bilinear transformations composed with the absolute value function. When acting on vectors, the TL1 penalty interpolates $l_0$ and $l_1$ similar to $l_p$ norm ($p \in (0,1)$). In our companion paper, we showed that TL1 is a robust sparsity promoting penalty in compressed sensing (CS) problems for a broad range of incoherent and coherent sensing matrices. Here we develop an explicit fixed point representation for the TL1 regularized minimization problem. The TL1 thresholding functions are in closed form for all parameter values. In contrast, the $l_p$ thresholding functions ($p \in [0,1]$) are in closed form only for $p=0,1,1/2,2/3$, known as hard, soft, half, and 2/3 thresholding respectively. The TL1 threshold values differ in subcritical (supercritical) parameter regime where the TL1 threshold functions are continuous (discontinuous) similar to soft-thresholding (half-thresholding) functions. We propose TL1 iterative thresholding algorithms and compare them with hard and half thresholding algorithms in CS test problems. For both incoherent and coherent sensing matrices, a proposed TL1 iterative thresholding algorithm with adaptive subcritical and supercritical thresholds consistently performs the best in sparse signal recovery with and without measurement noise.

cs.IT↗

arXiv2016

Transformed Schatten-1 Iterative Thresholding Algorithms for Low Rank Matrix Completion

We study a non-convex low-rank promoting penalty function, the transformed Schatten-1 (TS1), and its applications in matrix completion. The TS1 penalty, as a matrix quasi-norm defined on its singular values, interpolates the rank and the nuclear norm through a nonnegative parameter a. We consider the unconstrained TS1 regularized low-rank matrix recovery problem and develop a fixed point representation for its global minimizer. The TS1 thresholding functions are in closed analytical form for all parameter values. The TS1 threshold values differ in subcritical (supercritical) parameter regime where the TS1 threshold functions are continuous (discontinuous). We propose TS1 iterative thresholding algorithms and compare them with some state-of-the-art algorithms on matrix completion test problems. For problems with known rank, a fully adaptive TS1 iterative thresholding algorithm consistently performs the best under different conditions with ground truth matrix being multivariate Gaussian at varying covariance. For problems with unknown rank, TS1 algorithms with an additional rank estimation procedure approach the level of IRucL-q which is an iterative reweighted algorithm, non-convex in nature and best in performance.

cs.IT↗

arXiv2016

Ballistic Orbits and Front Speed Enhancement for ABC Flows

We study the two main types of trajectories of the ABC flow in the near-integrable regime: spiral orbits and edge orbits. The former are helical orbits which are perturbations of similar orbits that exist in the integrable regime, while the latter exist only in the non-integrable regime. We prove existence of ballistic (i.e., linearly growing) spiral orbits by using the contraction mapping principle in the Hamiltonian formulation, and we also find and analyze ballistic edge orbits. We discuss the relationship of existence of these orbits with questions concerning front propagation in the presence of flows, in particular, the question of linear (i.e., maximal possible) front speed enhancement rate for ABC flows.

math.DS↗

arXiv2016

Sparse Kalman Filtering Approaches to Covariance Estimation from High Frequency Data in the Presence of Jumps

Estimation of the covariance matrix of asset returns from high frequency data is complicated by asynchronous returns, market mi- crostructure noise and jumps. One technique for addressing both asynchronous returns and market microstructure is the Kalman-EM (KEM) algorithm. However the KEM approach assumes log-normal prices and does not address jumps in the return process which can corrupt estimation of the covariance matrix. In this paper we extend the KEM algorithm to price models that include jumps. We propose two sparse Kalman filtering approaches to this problem. In the first approach we develop a Kalman Expectation Conditional Maximization (KECM) algorithm to determine the un- known covariance as well as detecting the jumps. For this algorithm we consider Laplace and the spike and slab jump models, both of which promote sparse estimates of the jumps. In the second method we take a Bayesian approach and use Gibbs sampling to sample from the posterior distribution of the covariance matrix under the spike and slab jump model. Numerical results using simulated data show that each of these approaches provide for improved covariance estima- tion relative to the KEM method in a variety of settings where jumps occur.

q-fin.ST↗

arXiv2016

Periodic orbits of the ABC flow with $A=B=C=1$

In this paper, we prove that the ODE system $$ \begin{align*} \dot x &=\sin z+\cos y\\ \dot y &= \sin x+\cos z\\ \dot z &=\sin y + \cos x, \end{align*} $$ whose right-hand side is the Arnold-Beltrami-Childress (ABC) flow with parameters $A=B=C=1$, has periodic orbits on $(2π\mathbb T)^3$ with rotation vectors parallel to $(1,0,0)$, $(0,1,0)$, and $(0,0,1)$. An application of this result is that the well-known G-equation model for turbulent combustion with this ABC flow on $\mathbb R^3$ has a linear (i.e., maximal possible) flame speed enhancement rate as the amplitude of the flow grows.

math.AP↗

arXiv2015

Weighted Elastic Net Penalized Mean-Variance Portfolio Design and Computation

It is well known that the out-of-sample performance of Markowitz's mean-variance portfolio criterion can be negatively affected by estimation errors in the mean and covariance. In this paper we address the problem by regularizing the mean-variance objective function with a weighted elastic net penalty. We show that the use of this penalty can be motivated by a robust reformulation of the mean-variance criterion that directly accounts for parameter uncertainty. With this interpretation of the weighted elastic net penalty we derive data driven techniques for calibrating the weighting parameters based on the level of uncertainty in the parameter estimates. We test our proposed technique on US stock return data and our results show that the calibrated weighted elastic net penalized portfolio outperforms both the unpenalized portfolio and uniformly weighted elastic net penalized portfolio. This paper also introduces a novel Adaptive Support Split-Bregman approach which leverages the sparse nature of $\ell_{1}$ penalized portfolios to efficiently compute a solution of our proposed portfolio criterion. Numerical results show that this modification to the Split-Bregman algorithm results in significant improvements in computational speed compared with other techniques.

q-fin.PM↗