arXiv subjects

J. Zico Kolter

Publications and source records attributed to J. Zico Kolter.

169 records · Page 10Linked to original sources

arXiv2017

A Semismooth Newton Method for Fast, Generic Convex Programming

We introduce Newton-ADMM, a method for fast conic optimization. The basic idea is to view the residuals of consecutive iterates generated by the alternating direction method of multipliers (ADMM) as a set of fixed point equations, and then use a nonsmooth Newton method to find a solution; we apply the basic idea to the Splitting Cone Solver (SCS), a state-of-the-art method for solving generic conic optimization problems. We demonstrate theoretically, by extending the theory of semismooth operators, that Newton-ADMM converges rapidly (i.e., quadratically) to a solution; empirically, Newton-ADMM is significantly faster than SCS on a number of problems. The method also has essentially no tuning parameters, generates certificates of primal or dual infeasibility, when appropriate, and can be specialized to solve specific convex problems.

math.OC↗

arXiv2017

Input Convex Neural Networks

This paper presents the input convex neural network architecture. These are scalar-valued (potentially deep) neural networks with constraints on the network parameters such that the output of the network is a convex function of (some of) the inputs. The networks allow for efficient inference via optimization over some inputs to the network given others, and can be applied to settings including structured prediction, data imputation, reinforcement learning, and others. In this paper we lay the basic groundwork for these models, proposing methods for inference, optimization and learning, and analyze their representational power. We show that many existing neural network architectures can be made input-convex with a minor modification, and develop specialized optimization algorithms tailored to this setting. Finally, we highlight the performance of the methods on multi-label prediction, image completion, and reinforcement learning problems, where we show improvement over the existing state of the art in many cases.

cs.LG↗

arXiv2016

The Multiple Quantile Graphical Model

We introduce the Multiple Quantile Graphical Model (MQGM), which extends the neighborhood selection approach of Meinshausen and Buhlmann for learning sparse graphical models. The latter is defined by the basic subproblem of modeling the conditional mean of one variable as a sparse function of all others. Our approach models a set of conditional quantiles of one variable as a sparse function of all others, and hence offers a much richer, more expressive class of conditional distribution estimates. We establish that, under suitable regularity conditions, the MQGM identifies the exact conditional independencies with probability tending to one as the problem size grows, even outside of the usual homoskedastic Gaussian data model. We develop an efficient algorithm for fitting the MQGM using the alternating direction method of multipliers. We also describe a strategy for sampling from the joint distribution that underlies the MQGM estimate. Lastly, we present detailed experiments that demonstrate the flexibility and effectiveness of the MQGM in modeling hetereoskedastic non-Gaussian data.

stat.ME↗

arXiv2015

Convex programming with fast proximal and linear operators

We present Epsilon, a system for general convex programming using fast linear and proximal operators. As with existing convex programming frameworks, users specify convex optimization problems using a natural grammar for mathematical expressions, composing functions in a way that is guaranteed to be convex by the rules of disciplined convex programming. Given such an input, the Epsilon compiler transforms the optimization problem into a mathematically equivalent form consisting only of functions with efficient proximal operators---an intermediate representation we refer to as prox-affine form. By reducing problems to this form, Epsilon enables solving general convex problems using a large library of fast proximal and linear operators; numerical examples on many popular problems from statistics and machine learning show that this often improves running times by an order of magnitude or more vs. existing approaches based on conic solvers.

math.OC↗

arXiv2015

Probabilistic Segmentation via Total Variation Regularization

We present a convex approach to probabilistic segmentation and modeling of time series data. Our approach builds upon recent advances in multivariate total variation regularization, and seeks to learn a separate set of parameters for the distribution over the observations at each time point, but with an additional penalty that encourages the parameters to remain constant over time. We propose efficient optimization methods for solving the resulting (large) optimization problems, and a two-stage procedure for estimating recurring clusters under such models, based upon kernel density estimation. Finally, we show on a number of real-world segmentation tasks, the resulting methods often perform as well or better than existing latent variable models, while being substantially easier to train.

stat.ML↗

arXiv2013

Contextually Supervised Source Separation with Application to Energy Disaggregation

We propose a new framework for single-channel source separation that lies between the fully supervised and unsupervised setting. Instead of supervision, we provide input features for each source signal and use convex methods to estimate the correlations between these features and the unobserved signal decomposition. We analyze the case of $\ell_2$ loss theoretically and show that recovery of the signal components depends only on cross-correlation between features for different signals, not on correlations between features for the same signal. Contextually supervised source separation is a natural fit for domains with large amounts of data but no explicit supervision; our motivating application is energy disaggregation of hourly smart meter data (the separation of whole-home power signals into different energy uses). Here we apply contextual supervision to disaggregate the energy usage of thousands homes over four years, a significantly larger scale than previously published efforts, and demonstrate on synthetic data that our method outperforms the unsupervised approach.

stat.ML↗

arXiv2013

A Fast Algorithm for Sparse Controller Design

We consider the task of designing sparse control laws for large-scale systems by directly minimizing an infinite horizon quadratic cost with an $\ell_1$ penalty on the feedback controller gains. Our focus is on an improved algorithm that allows us to scale to large systems (i.e. those where sparsity is most useful) with convergence times that are several orders of magnitude faster than existing algorithms. In particular, we develop an efficient proximal Newton method which minimizes per-iteration cost with a coordinate descent active set approach and fast numerical solutions to the Lyapunov equations. Experimentally we demonstrate the appeal of this approach on synthetic examples and real power networks significantly larger than those previously considered in the literature.

math.OC↗