arXiv subjects

Tong Zhang

Publications and source records attributed to Tong Zhang.

At least 649 records · Page 36Linked to original sources

arXiv2017

Second-Order Methods with Cubic Regularization Under Inexact Information

In this paper, we generalize (accelerated) Newton's method with cubic regularization under inexact second-order information for (strongly) convex optimization problems. Under mild assumptions, we provide global rate of convergence of these methods and show the explicit dependence of the rate of convergence on the problem parameters. While the complexity bounds of our presented algorithms are theoretically worse than those of their exact counterparts, they are at least as good as those of the optimal first-order methods. Our numerical experiments also show that using inexact Hessians can significantly speed up the algorithms in practice.

math.OC↗

arXiv2017

Improved Optimization of Finite Sums with Minibatch Stochastic Variance Reduced Proximal Iterations

We present novel minibatch stochastic optimization methods for empirical risk minimization problems, the methods efficiently leverage variance reduced first-order and sub-sampled higher-order information to accelerate the convergence speed. For quadratic objectives, we prove improved iteration complexity over state-of-the-art under reasonable assumptions. We also provide empirical evidence of the advantages of our method compared to existing approaches in the literature.

math.OC↗

arXiv2017

Near-Optimal Stochastic Approximation for Online Principal Component Estimation

Principal component analysis (PCA) has been a prominent tool for high-dimensional data analysis. Online algorithms that estimate the principal component by processing streaming data are of tremendous practical and theoretical interests. Despite its rich applications, theoretical convergence analysis remains largely open. In this paper, we cast online PCA into a stochastic nonconvex optimization problem, and we analyze the online PCA algorithm as a stochastic approximation iteration. The stochastic approximation iteration processes data points incrementally and maintains a running estimate of the principal component. We prove for the first time a nearly optimal finite-sample error bound for the online PCA algorithm. Under the subgaussian assumption, we show that the finite-sample error bound closely matches the minimax information lower bound.

math.OC↗

arXiv2017

Deep manifold-to-manifold transforming network for action recognition

Symmetric positive definite (SPD) matrices (e.g., covariances, graph Laplacians, etc.) are widely used to model the relationship of spatial or temporal domain. Nevertheless, SPD matrices are theoretically embedded on Riemannian manifolds. In this paper, we propose an end-to-end deep manifold-to-manifold transforming network (DMT-Net) which can make SPD matrices flow from one Riemannian manifold to another more discriminative one. To learn discriminative SPD features characterizing both spatial and temporal dependencies, we specifically develop three novel layers on manifolds: (i) the local SPD convolutional layer, (ii) the non-linear SPD activation layer, and (iii) the Riemannian-preserved recursive layer. The SPD property is preserved through all layers without any requirement of singular value decomposition (SVD), which is often used in the existing methods with expensive computation cost. Furthermore, a diagonalizing SPD layer is designed to efficiently calculate the final metric for the classification task. To evaluate our proposed method, we conduct extensive experiments on the task of action recognition, where input signals are popularly modeled as SPD matrices. The experimental results demonstrate that our DMT-Net is much more competitive over state-of-the-art.

cs.CV↗

arXiv2017

Deep Subspace Clustering Networks

We present a novel deep neural network architecture for unsupervised subspace clustering. This architecture is built upon deep auto-encoders, which non-linearly map the input data into a latent space. Our key idea is to introduce a novel self-expressive layer between the encoder and the decoder to mimic the "self-expressiveness" property that has proven effective in traditional subspace clustering. Being differentiable, our new self-expressive layer provides a simple but effective way to learn pairwise affinities between all data points through a standard back-propagation procedure. Being nonlinear, our neural-network based method is able to cluster data points having complex (often nonlinear) structures. We further propose pre-training and fine-tuning strategies that let us effectively learn the parameters of our subspace clustering networks. Our experiments show that the proposed method significantly outperforms the state-of-the-art unsupervised subspace clustering methods.

cs.CV↗

arXiv2017

A General Distributed Dual Coordinate Optimization Framework for Regularized Loss Minimization

In modern large-scale machine learning applications, the training data are often partitioned and stored on multiple machines. It is customary to employ the "data parallelism" approach, where the aggregated training loss is minimized without moving data across machines. In this paper, we introduce a novel distributed dual formulation for regularized loss minimization problems that can directly handle data parallelism in the distributed setting. This formulation allows us to systematically derive dual coordinate optimization procedures, which we refer to as Distributed Alternating Dual Maximization (DADM). The framework extends earlier studies described in (Boyd et al., 2011; Ma et al., 2015a; Jaggi et al., 2014; Yang, 2013) and has rigorous theoretical analyses. Moreover with the help of the new formulation, we develop the accelerated version of DADM (Acc-DADM) by generalizing the acceleration technique from (Shalev-Shwartz and Zhang, 2014) to the distributed setting. We also provide theoretical results for the proposed accelerated version and the new result improves previous ones (Yang, 2013; Ma et al., 2015a) whose runtimes grow linearly on the condition number. Our empirical studies validate our theory and show that our accelerated approach significantly improves the previous state-of-the-art distributed dual coordinate optimization algorithms.

cs.LG↗

arXiv2017

Graphical Nonconvex Optimization for Optimal Estimation in Gaussian Graphical Models

We consider the problem of learning high-dimensional Gaussian graphical models. The graphical lasso is one of the most popular methods for estimating Gaussian graphical models. However, it does not achieve the oracle rate of convergence. In this paper, we propose the graphical nonconvex optimization for optimal estimation in Gaussian graphical models, which is then approximated by a sequence of convex programs. Our proposal is computationally tractable and produces an estimator that achieves the oracle rate of convergence. The statistical error introduced by the sequential approximation using the convex programs are clearly demonstrated via a contraction property. The rate of convergence can be further improved using the notion of sparsity pattern. The proposed methodology is then extended to semiparametric graphical models. We show through numerical studies that the proposed estimator outperforms other popular methods for estimating Gaussian graphical models.

stat.ML↗

arXiv2017

Spatial-Temporal Recurrent Neural Network for Emotion Recognition

Emotion analysis is a crucial problem to endow artifact machines with real intelligence in many large potential applications. As external appearances of human emotions, electroencephalogram (EEG) signals and video face signals are widely used to track and analyze human's affective information. According to their common characteristics of spatial-temporal volumes, in this paper we propose a novel deep learning framework named spatial-temporal recurrent neural network (STRNN) to unify the learning of two different signal sources into a spatial-temporal dependency model. In STRNN, to capture those spatially cooccurrent variations of human emotions, a multi-directional recurrent neural network (RNN) layer is employed to capture longrange contextual cues by traversing the spatial region of each time slice from multiple angles. Then a bi-directional temporal RNN layer is further used to learn discriminative temporal dependencies from the sequences concatenating spatial features of each time slice produced from the spatial RNN layer. To further select those salient regions of emotion representation, we impose sparse projection onto those hidden states of spatial and temporal domains, which actually also increases the model discriminant ability because of this global consideration. Consequently, such a two-layer RNN model builds spatial dependencies as well as temporal dependencies of the input signals. Experimental results on the public emotion datasets of EEG and facial expression demonstrate the proposed STRNN method is more competitive over those state-of-the-art methods.

cs.CV↗

arXiv2017

Optimal Allocation of Power Pool for Two-Cell Single-User Joint Transmission

Joint transmission is a kind of cooperative transmission technology, which converts other cell interference into signal. In this letter, for single-user joint transmission, we propose that a power pool can be created in order to increase the system sum-rate or lower the power consumption. Then, we investigate the sum-rate maximization problem under this framework. Since it is impossible that subchannel gains for the two base stations are identical, we can obtain a much more concise structure and reduce variables to the half of original problem. And that the closed-form water-filling solution is unique and global optimum, because it is a convex optimization problem. Compared to the power allocation without pooling, the performance gain is verified by simulation.

cs.IT↗

arXiv2017

Power Allocation and Scheduling for Two-Cell Distributed Interference Subtraction System

This letter studies the power allocation and scheduling problem in two-cell distributed interference subtraction system. Through the interference is cancelled at one receiver side and the analysis of objective function monotony, our key result is the decoupling of power allocation and scheduling without loss of optimality. Simulation exhibits the significant gains of our proposed algorithm over open-loop power allocation with non-cooperation proportional fair scheduling.

cs.IT↗

arXiv2017

I-LAMM for Sparse Learning: Simultaneous Control of Algorithmic Complexity and Statistical Error

We propose a computational framework named iterative local adaptive majorize-minimization (I-LAMM) to simultaneously control algorithmic complexity and statistical error when fitting high dimensional models. I-LAMM is a two-stage algorithmic implementation of the local linear approximation to a family of folded concave penalized quasi-likelihood. The first stage solves a convex program with a crude precision tolerance to obtain a coarse initial estimator, which is further refined in the second stage by iteratively solving a sequence of convex programs with smaller precision tolerances. Theoretically, we establish a phase transition: the first stage has a sublinear iteration complexity, while the second stage achieves an improved linear rate of convergence. Though this framework is completely algorithmic, it provides solutions with optimal statistical performances and controlled algorithmic complexity for a large family of nonconvex optimization problems. The iteration effects on statistical errors are clearly demonstrated via a contraction property. Our theory relies on a localized version of the sparse/restricted eigenvalue condition, which allows us to analyze a large family of loss and penalty functions and provide optimality guarantees under very weak assumptions (For example, I-LAMM requires much weaker minimal signal strength than other procedures). Thorough numerical results are provided to support the obtained theory.

math.ST↗

arXiv2017

Pathwise Coordinate Optimization for Sparse Learning: Algorithm and Theory

The pathwise coordinate optimization is one of the most important computational frameworks for high dimensional convex and nonconvex sparse learning problems. It differs from the classical coordinate optimization algorithms in three salient features: {\it warm start initialization}, {\it active set updating}, and {\it strong rule for coordinate preselection}. Such a complex algorithmic structure grants superior empirical performance, but also poses significant challenge to theoretical analysis. To tackle this long lasting problem, we develop a new theory showing that these three features play pivotal roles in guaranteeing the outstanding statistical and computational performance of the pathwise coordinate optimization framework. Particularly, we analyze the existing pathwise coordinate optimization algorithms and provide new theoretical insights into them. The obtained insights further motivate the development of several modifications to improve the pathwise coordinate optimization framework, which guarantees linear convergence to a unique sparse local optimum with optimal statistical properties in parameter estimation and support recovery. This is the first result on the computational and statistical guarantees of the pathwise coordinate optimization framework in high dimensions. Thorough numerical experiments are provided to support our theory.

stat.ML↗

arXiv2017

Multi-scale Convolutional Neural Networks for Crowd Counting

Crowd counting on static images is a challenging problem due to scale variations. Recently deep neural networks have been shown to be effective in this task. However, existing neural-networks-based methods often use the multi-column or multi-network model to extract the scale-relevant features, which is more complicated for optimization and computation wasting. To this end, we propose a novel multi-scale convolutional neural network (MSCNN) for single image crowd counting. Based on the multi-scale blobs, the network is able to generate scale-relevant features for higher crowd counting performances in a single-column architecture, which is both accuracy and cost effective for practical applications. Complemental results show that our method outperforms the state-of-the-art methods on both accuracy and robustness with far less number of parameters.

cs.CV↗

arXiv2016

Towards More Efficient SPSD Matrix Approximation and CUR Matrix Decomposition

Symmetric positive semi-definite (SPSD) matrix approximation methods have been extensively used to speed up large-scale eigenvalue computation and kernel learning methods. The standard sketch based method, which we call the prototype model, produces relatively accurate approximations, but is inefficient on large square matrices. The Nyström method is highly efficient, but can only achieve low accuracy. In this paper we propose a novel model that we call the {\it fast SPSD matrix approximation model}. The fast model is nearly as efficient as the Nyström method and as accurate as the prototype model. We show that the fast model can potentially solve eigenvalue problems and kernel learning problems in linear time with respect to the matrix size $n$ to achieve $1+ε$ relative-error, whereas both the prototype model and the Nyström method cost at least quadratic time to attain comparable error bound. Empirical comparisons among the prototype model, the Nyström method, and our fast model demonstrate the superiority of the fast model. We also contribute new understandings of the Nyström method. The Nyström method is a special instance of our fast model and is approximation to the prototype model. Our technique can be straightforwardly applied to make the CUR matrix decomposition more efficiently computed without much affecting the accuracy.

cs.LG↗

arXiv2016

Chromatic Effect for THz Generation in a Novel Wave-front Tilt Scheme

Deriving single or few cycle terahertz pulse (THz) by intense femtosecond laser through cascaded optical rectification in electro-optic crystals is a crucial technique in cutting-edge time-resolved spectroscopy to characterize micro-scale structures and ultrafast dynamics. In the past decade, lithium niobate (LN) crystal implementation of wave-front tilt scheme has been prevalently used, while painstaking efforts have been invested in order to achieve higher THz conversion efficiency. In this research we developed a brand new type of LN crystal possessing dual-face-cut and Brewster coupling, and conducted experimental and simulative investigation systematically to optimize the multi-dimensionally entangled parameters in THz generation, predicting the extreme conversion efficiency of 10% is potentially promising at the THz absorption coefficient of 0.5cm-1. More remarkably, we first discovered that the chirp of the driving laser pulse plays a decisive role in the wave-front tilt scheme, and the THz generation efficiency could be enhanced tremendously by applying an appropriate chirp.

physics.optics↗

arXiv2016

Perfect Memory Context Trees in time series modeling

The Stochastic Context Tree (SCOT) is a useful tool for studying infinite random sequences generated by an m-Markov Chain (m-MC). It captures the phenomenon that the probability distribution of the next state sometimes depends on less than m of the preceding states. This allows compressing the information needed to describe an m-MC. The SCOT construction has been earlier used under various names: VLMC, VOMC, PST, CTW. In this paper we study the possibility of reducing the m-MC to a 1-MC on the leaves of the SCOT. Such context trees are called perfect-memory. We give various combinatorial characterizations of perfect-memory context trees and an efficient algorithm to find the minimal perfect-memory extension of a SCOT.

cs.LO↗

arXiv2016

Convolutional Neural Networks for Text Categorization: Shallow Word-level vs. Deep Character-level

This paper reports the performances of shallow word-level convolutional neural networks (CNN), our earlier work (2015), on the eight datasets with relatively large training data that were used for testing the very deep character-level CNN in Conneau et al. (2016). Our findings are as follows. The shallow word-level CNNs achieve better error rates than the error rates reported in Conneau et al., though the results should be interpreted with some consideration due to the unique pre-processing of Conneau et al. The shallow word-level CNN uses more parameters and therefore requires more storage than the deep character-level CNN; however, the shallow word-level CNN computes much faster.

cs.CL↗

arXiv2016

Slope inequality for families of curves over surfaces

In this paper, we investigate the general notion of the slope for families of curves $f: X \to Y$. The main result is an answer to the above question when $\dim Y = 2$, and we prove a lower bound for this new slope in this case over fields of any characteristic. Both the notion and the slope inequality are compatible with the theory for $\dim Y = 0, 1$ in a very natural way, and this gives a strong evidence that the slope for an $n$-fold fibration of curves $f: X \to Y$ may be $K_{X/Y}^n / \mathrm{ch}_{n-1}(f_* ω_{X/Y})$. Rather than the usual stability methods, the whole proof of the slope inequality here is based on a completely new method using characteristic $p>0$ geometry. A simpler version of this method yields a new proof of the slope inequality when $\dim Y = 1$.

math.AG↗