arXiv subjects

Adam Oberman

Publications and source records attributed to Adam Oberman.

29 records · Page 2Linked to original sources

arXiv2019

Lipschitz regularized Deep Neural Networks generalize and are adversarially robust

In this work we study input gradient regularization of deep neural networks, and demonstrate that such regularization leads to generalization proofs and improved adversarial robustness. The proof of generalization does not overcome the curse of dimensionality, but it is independent of the number of layers in the networks. The adversarial robustness regularization combines adversarial training, which we show to be equivalent to Total Variation regularization, with Lipschitz regularization. We demonstrate empirically that the regularized models are more robust, and that gradient norms of images can be used for attack detection.

cs.LG↗

arXiv2019

A Partial Differential Equation Obstacle Problem for the Level Set Approach to Visibility

In this article we consider the problem of finding the visibility set from a given point when the obstacles are represented as the level set of a given function. Although the visibility set can be computed efficiently by ray tracing, there are advantages to using a level set representation for the obstacles, and to characterizing the solution using a Partial Differential Equation (PDE). A nonlocal PDE formulation was proposed in Tsai et. al. (Journal of Computational Physics 199(1):260-290, 2004): in this article we propose a simpler PDE formulation, involving a nonlinear obstacle problem. We present a simple numerical scheme and show its convergence using the framework of Barles and Souganidis. Numerical examples in both two and three dimensions are presented.

math.AP↗

arXiv2018

Improved accuracy of monotone finite difference schemes on point clouds and regular grids

Finite difference schemes are the method of choice for solving nonlinear, degenerate elliptic PDEs, because the Barles-Sougandis convergence framework [Barles and Sougandidis, Asymptotic Analysis, 4(3):271-283, 1991] provides sufficient conditions for convergence to the unique viscosity solution [Crandall, Ishii and Lions, Bull. Amer. Math Soc., 27(1):1-67, 1992]. For anisotropic operators, such as the Monge-Ampere equation, wide stencil schemes are needed [Oberman, SIAM J. Numer. Anal., 44(2):879-895]. The accuracy of these schemes depends on both the distances to neighbors, $R$, and the angular resolution, $d\theta$. On uniform grids, the accuracy is $\mathcal O(R^2 + d\theta)$. On point clouds, the most accurate schemes are of $\mathcal O(R + d\theta)$, by Froese [Numerische Mathematik, 138(1):75-99, 2018]. In this work, we construct geometrically motivated schemes of higher accuracy in both cases: order $\mathcal O(R + d\theta^2)$ on point clouds, and $\mathcal O(R^2 + d\theta^2)$ on uniform grids.

math.NA↗

arXiv2018

Stochastic Backward Euler: An Implicit Gradient Descent Algorithm for $k$-means Clustering

In this paper, we propose an implicit gradient descent algorithm for the classic $k$-means problem. The implicit gradient step or backward Euler is solved via stochastic fixed-point iteration, in which we randomly sample a mini-batch gradient in every iteration. It is the average of the fixed-point trajectory that is carried over to the next gradient step. We draw connections between the proposed stochastic backward Euler and the recent entropy stochastic gradient descent (Entropy-SGD) for improving the training of deep neural networks. Numerical experiments on various synthetic and real datasets show that the proposed algorithm provides better clustering results compared to $k$-means algorithms in the sense that it decreased the objective function (the cluster) and is much more robust to initialization.

math.OC↗

arXiv2017

Parle: parallelizing stochastic gradient descent

We propose a new algorithm called Parle for parallel training of deep networks that converges 2-4x faster than a data-parallel implementation of SGD, while achieving significantly improved error rates that are nearly state-of-the-art on several benchmarks including CIFAR-10 and CIFAR-100, without introducing any additional hyper-parameters. We exploit the phenomenon of flat minima that has been shown to lead to improved generalization error for deep networks. Parle requires very infrequent communication with the parameter server and instead performs more computation on each client, which makes it well-suited to both single-machine, multi-GPU settings and distributed implementations.

cs.LG↗

arXiv2017

Deep Relaxation: partial differential equations for optimizing deep neural networks

In this paper we establish a connection between non-convex optimization methods for training deep neural networks and nonlinear partial differential equations (PDEs). Relaxation techniques arising in statistical physics which have already been used successfully in this context are reinterpreted as solutions of a viscous Hamilton-Jacobi PDE. Using a stochastic control interpretation allows we prove that the modified algorithm performs better in expectation that stochastic gradient descent. Well-known PDE regularity results allow us to analyze the geometry of the relaxed energy landscape, confirming empirical evidence. The PDE is derived from a stochastic homogenization problem, which arises in the implementation of the algorithm. The algorithms scale well in practice and can effectively tackle the high dimensionality of modern neural networks.

cs.LG↗

arXiv2017

Anisotropic Total Variation Regularized L^1-Approximation and Denoising/Deblurring of 2D Bar Codes

We consider variations of the Rudin-Osher-Fatemi functional which are particularly well-suited to denoising and deblurring of 2D bar codes. These functionals consist of an anisotropic total variation favoring rectangles and a fidelity term which measure the L^1 distance to the signal, both with and without the presence of a deconvolution operator. Based upon the existence of a certain associated vector field, we find necessary and sufficient conditions for a function to be a minimizer. We apply these results to 2D bar codes to find explicit regimes ---in terms of the fidelity parameter and smallest length scale of the bar codes--- for which a perfect bar code is recoverable via minimization of the functionals. Via a discretization reformulated as a linear program, we perform numerical experiments for all functionals demonstrating their denoising and deblurring capabilities.

math.OC↗

arXiv2016

Finite difference methods for fractional Laplacians

The fractional Laplacian $(-\Delta)^{\alpha/2}$ is the prototypical non-local elliptic operator. While analytical theory has been advanced and understood for some time, there remain many open problems in the numerical analysis of the operator. In this article, we study several different finite difference discretisations of the fractional Laplacian on uniform grids in one dimension that takes the same form. Many properties can be compared and summarised in this relatively simple setting, to tackle more important questions like the nonlocality, singularity and flat tails common in practical implementations. The accuracy and the asymptotic behaviours of the methods are also studied, together with treatment of the far field boundary conditions, providing a unified perspective on the further development of the scheme in higher dimensions.

math.NA↗

arXiv2014

Numerical methods for matching for teams and Wasserstein barycenters

Equilibrium multi-population matching (matching for teams) is a problem from mathematical economics which is related to multi-marginal optimal transport. A special but important case is the Wasserstein barycenter problem, which has applications in image processing and statistics. Two algorithms are presented: a linear programming algorithm and an efficient nonsmooth optimization algorithm, which applies in the case of the Wasserstein barycenters. The measures are approximated by discrete measures: convergence of the approximation is proved. Numerical results are presented which illustrate the efficiency of the algorithms.

math.NA↗

arXiv2013

Numerical Methods for the Fractional Laplacian: a Finite Difference-quadrature Approach

The fractional Laplacian $(-\Delta)^{\alpha/2}$ is a non-local operator which depends on the parameter $\alpha$ and recovers the usual Laplacian as $\alpha \to 2$. A numerical method for the fractional Laplacian is proposed, based on the singular integral representation for the operator. The method combines finite difference with numerical quadrature, to obtain a discrete convolution operator with positive weights. The accuracy of the method is shown to be $O(h^{3-\alpha})$. Convergence of the method is proven. The treatment of far field boundary conditions using an asymptotic approximation to the integral is used to obtain an accurate method. Numerical experiments on known exact solutions validate the predicted convergence rates. Computational examples include exponentially and algebraically decaying solution with varying regularity. The generalization to nonlinear equations involving the operator is discussed: the obstacle problem for the fractional Laplacian is computed.

math.NA↗

arXiv1999

Bulk Burning Rate in Passive - Reactive Diffusion

We consider a passive scalar that is advected by a prescribed mean zero divergence-free velocity field, diffuses, and reacts according to a KPP-type nonlinear reaction. We introduce a quantity, the bulk burning rate, that makes both mathematical and physical sense in general situations and extends the often ill-defined notion of front speed. We establish rigorous lower bounds for the bulk burning rate that are linear in the amplitude of the advecting velocity for a large class of flows. These "percolating" flows are characterized by the presence of tubes of streamlines connecting distant regions of burned and unburned material and generalize shear flows. The bound contains geometric information on the velocity streamlines and degenerates when these oscillate on scales that are finer than the width of the laminar burning region. We give also examples of very different kind of flows, cellular flows with closed streamlines, and rigorously prove that these can produce only sub-linear enhancement of the bulk burning rate.

math.AP↗