arXiv subjects

Kaj Nyström

Publications and source records attributed to Kaj Nyström.

At least 37 records · Page 2Linked to original sources

arXiv2022

Deep learning, stochastic gradient descent and diffusion maps

Stochastic gradient descent (SGD) is widely used in deep learning due to its computational efficiency, but a complete understanding of why SGD performs so well remains a major challenge. It has been observed empirically that most eigenvalues of the Hessian of the loss functions on the loss landscape of over-parametrized deep neural networks are close to zero, while only a small number of eigenvalues are large. Zero eigenvalues indicate zero diffusion along the corresponding directions. This indicates that the process of minima selection mainly happens in the relatively low-dimensional subspace corresponding to the top eigenvalues of the Hessian. Although the parameter space is very high-dimensional, these findings seems to indicate that the SGD dynamics may mainly live on a low-dimensional manifold. In this paper, we pursue a truly data driven approach to the problem of getting a potentially deeper understanding of the high-dimensional parameter surface, and in particular, of the landscape traced out by SGD by analyzing the data generated through SGD, or any other optimizer for that matter, in order to possibly discover (local) low-dimensional representations of the optimization landscape. As our vehicle for the exploration, we use diffusion maps introduced by R. Coifman and coauthors.

stat.ML↗

arXiv2022

On regularity and existence of weak solutions to nonlinear Kolmogorov-Fokker-Planck type equations with rough coefficients

We consider nonlinear Kolmogorov-Fokker-Planck type equations of the form \begin{equation}\label{abeqn} (\partial_t+X\cdot\nabla_Y)u=\nabla_X\cdot(A(\nabla_X u,X,Y,t)). \end{equation} The function $A=A(ξ,X,Y,t):\R^m\times\R^m\times\R^m\times\R\to\R^m$ is assumed to be continuous with respect to $ξ$, and measurable with respect to $X,Y$ and $t$. $A=A(ξ,X,Y,t)$ is allowed to be nonlinear but with linear growth. We establish higher integrability and local boundedness of weak sub-solutions, weak Harnack and Harnack inequalities, and H{ö}lder continuity with quantitative estimates. In addition we establish existence and uniqueness of weak solutions to a Dirichlet problem in certain bounded $X$, $Y$ and $t$ dependent domains.

math.AP↗

arXiv2022

Asymptotic mean value formulas, nonlocal space-time parabolic operators and anomalous tug-of-war games

The fractional heat operator $(\partial_t-Δ_x)^s$ and Continuous Time Random Walks (CTRWs) are interesting and sophisticated mathematical models that can describe complex anomalous systems. In this paper, we prove asymptotic mean value representation formulas for functions with respect to $(\partial_t-Δ_x)^s$ and we introduce new nonlocal, nonlinear parabolic operators related to a tug-of-war which accounts for waiting times and space-time couplings. These nonlocal, nonlinear parabolic operators and equations can be seen as nonlocal versions of the evolutionary infinity Laplace operator.

math.AP↗

arXiv2021

$L^2$ well-posedness of boundary value problems for parabolic systems with measurable coefficients

We prove the first positive results concerning boundary value problems in the upper half-space of second order parabolic systems only assuming measurability and some transversal regularity in the coefficients of the elliptic part. To do so, we introduce and develop a first order strategy by means of a parabolic Dirac operator at the boundary to obtain, in particular, Green's representation for solutions in natural classes involving square functions and non-tangential maximal functions, well-posedness results with data in $L^2$-Sobolev spaces together with invertibility of layer potentials, and perturbation results. In the way, we solve the Kato square root problem for parabolic operators with coefficients of the elliptic part depending measurably on all variables. The major new challenge, compared to the earlier results by one of us under time and transversally independence of the coefficients, is to handle non-local half-order derivatives in time which are unavoidable in our situation.

math.CA↗

arXiv2021

The Dirichlet problem for second order parabolic operators in divergence form

We study parabolic operators H = $\partial$t -- div $λ$,x A(x, t)$\nabla$ $λ$,x in the parabolic upper half space R n+2 + = {($λ$, x, t) : $λ$ > 0}. We assume that the coefficients are real, bounded, measurable, uniformly elliptic, but not necessarily symmetric. We prove that the associated parabolic measure is absolutely continuous with respect to the surface measure on R n+1 in the sense defined by A$\infty$(dx dt). Our argument also gives a simplified proof of the corresponding result for elliptic measure.

math.AP↗

arXiv2021

Carleson measure estimates for caloric functions and parabolic uniformly rectifiable sets

Let $E \subset \mathbb R^{n+1}$ be a parabolic uniformly rectifiable set. We prove that every bounded solution $u$ to $$\partial_tu- Δu=0, \quad \text{in} \quad \mathbb R^{n+1}\setminus E$$ satisfies a Carleson measure estimate condition. An important technical novelty of our work is that we develop a corona domain approximation scheme for $E$ in terms of regular Lip(1/2,1) graph domains. This approximation scheme has an analogous elliptic version which is an improvement of the known results in that setting.

math.AP↗

arXiv2021

On Big Pieces approximations of parabolic hypersurfaces

Let $Σ$ be a closed subset of $\mathbb{R}^ {n+1}$ which is parabolic Ahlfors-David regular and assume that $Σ$ satisfies a 2-sided corkscrew condition. Assume, in addition, that $Σ$ is either time-forwards Ahlfors-David regular, time-backwards Ahlfors-David regular, or parabolic uniform rectifiable. We then first prove that $Σ$ satisfies a {\it weak synchronized two cube condition}. Based on this we are able to revisit the argument in \cite{NS} and prove that $Σ$ contains {\it uniform big pieces of Lip(1,1/2) graphs}. When $Σ$ is parabolic uniformly rectifiable the construction can be refined and in this case we prove that $Σ$ contains {\it uniform big pieces of regular parabolic Lip(1,1/2) graphs}. Similar results hold if $Ω\subset\mathbb R^{n+1}$ is a connected component of $\mathbb R^{n+1}\setminusΣ$ and in this context we also give a parabolic counterpart of the main result in \cite{AHMNT} by proving that if $Ω$ is a one-sided parabolic chord arc domain, and if $Σ$ is parabolic uniformly rectifiable, then $Ω$ is in fact a parabolic chord arc domain. Our results give a flexible parabolic version of the classical (elliptic) result of G. David and D. Jerison concerning the existence of uniform big pieces of Lipschitz graphs for sets satisfying a two disc condition.

math.AP↗

arXiv2020

Coronizations and big pieces in metric spaces

We prove that coronizations with respect to arbitrary d-regular sets (not necessarily graphs) imply big pieces squared of these (approximating) sets. This is known (and due to David and Semmes in the case of sufficiently large co-dimension, and to Azzam and Schul in general) in the (classical) setting of Euclidean spaces with Hausdorff measure of integer dimension, where the approximating sets are Lipschitz graphs. Our result is a far reaching generalization of these results and we prove that coronizations imply big pieces squared is a generic property. In particular, our result applies, when suitably interpreted, in metric spaces having a fixed positive (perhaps non-integer) dimension, equipped with a Borel regular measure and with arbitrary approximating sets. As a novel application we highlight how to utilize this general setting in the context of parabolic uniform rectifiability.

math.CA↗

arXiv2020

Neural ODEs as the Deep Limit of ResNets with constant weights

In this paper we prove that, in the deep limit, the stochastic gradient descent on a ResNet type deep neural network, where each layer shares the same weight matrix, converges to the stochastic gradient descent for a Neural ODE and that the corresponding value/loss functions converge. Our result gives, in the context of minimization by stochastic gradient descent, a theoretical foundation for considering Neural ODEs as the deep limit of ResNets. Our proof is based on certain decay estimates for associated Fokker-Planck equations.

stat.ML↗

arXiv2018

Neural network augmented inverse problems for PDEs

In this paper we show how to augment classical methods for inverse problems with artificial neural networks. The neural network acts as a prior for the coefficient to be estimated from noisy data. Neural networks are global, smooth function approximators and as such they do not require explicit regularization of the error functional to recover smooth solutions and coefficients. We give detailed examples using the Poisson equation in 1, 2, and 3 space dimensions and show that the neural network augmentation is robust with respect to noisy and incomplete data, mesh, and geometry.

stat.ML↗

arXiv2018

Data-driven discovery of PDEs in complex datasets

Many processes in science and engineering can be described by partial differential equations (PDEs). Traditionally, PDEs are derived by considering first principles of physics to derive the relations between the involved physical quantities of interest. A different approach is to measure the quantities of interest and use deep learning to reverse engineer the PDEs which are describing the physical process. In this paper we use machine learning, and deep learning in particular, to discover PDEs hidden in complex data sets from measurement data. We include examples of data from a known model problem, and real data from weather station measurements. We show how necessary transformations of the input data amounts to coordinate transformations in the discovered PDE, and we elaborate on feature and model selection. It is shown that the dynamics of a non-linear, second order PDE can be accurately described by an ordinary differential equation which is automatically discovered by our deep learning algorithm. Even more interestingly, we show that similar results apply in the context of more complex simulations of the Swedish temperature distribution.

stat.ML↗

arXiv2018

A unified deep artificial neural network approach to partial differential equations in complex geometries

In this paper we use deep feedforward artificial neural networks to approximate solutions to partial differential equations in complex geometries. We show how to modify the backpropagation algorithm to compute the partial derivatives of the network output with respect to the space variables which is needed to approximate the differential operator. The method is based on an ansatz for the solution which requires nothing but feedforward neural networks and an unconstrained gradient based optimization method such as gradient descent or a quasi-Newton method. We show an example where classical mesh based methods cannot be used and neural networks can be seen as an attractive alternative. Finally, we highlight the benefits of deep compared to shallow neural networks and device some other convergence enhancing techniques.

stat.ML↗

arXiv2016

Boundedness of single layer potentials associated to divergence form parabolic equations with complex coefficients

We consider parabolic operators of the form $$\partial_t+\mathcal{L},\ \mathcal{L}:=-\mbox{div}\, A(X,t)\nabla,$$ in $\mathbb R_+^{n+2}:=\{(X,t)=(x,x_{n+1},t)\in \mathbb R^{n}\times \mathbb R\times \mathbb R:\ x_{n+1}>0\}$, $n\geq 1$. We assume that $A$ is a $(n+1)\times (n+1)$-dimensional matrix which is bounded, measurable, uniformly elliptic and complex, and we assume, in addition, that the entries of A are independent of the spatial coordinate $x_{n+1}$ as well as of the time coordinate $t$. We prove that the boundedness of associated single layer potentials, with data in $L^2$, can be reduced to two crucial estimates, one being a square function estimate involving the single layer potential. By establishing a local parabolic Tb-theorem for square functions we are then able to verify the two crucial estimates in the case of real, symmetric operators. As part of this argument we establish a scale-invariant reverse H{ö}lder inequality for the parabolic Poisson kernel. Our results are important when addressing the solvability of the classical Dirichlet, Neumann and Regularity problems for the operator $\partial_t+\mathcal{L}$ in $\mathbb R_+^{n+2}$, with $L^2$-data on $\mathbb R^{n+1}=\partial\mathbb R_+^{n+2}$, and by way of layer potentials.

math.AP↗

arXiv2016

$L^2$ Solvability of boundary value problems for divergence form parabolic equations with complex coefficients

We consider parabolic operators of the form $$\partial_t+\mathcal{L},\ \mathcal{L}=-\mbox{div}\, A(X,t)\nabla,$$ in $\mathbb R_+^{n+2}:=\{(X,t)=(x,x_{n+1},t)\in \mathbb R^{n}\times \mathbb R\times \mathbb R:\ x_{n+1}>0\}$, $n\geq 1$. We assume that $A$ is a $(n+1)\times (n+1)$-dimensional matrix which is bounded, measurable, uniformly elliptic and complex, and we assume, in addition, that the entries of A are independent of the spatial coordinate $x_{n+1}$ as well as of the time coordinate $t$. For such operators we prove that the boundedness and invertibility of the corresponding layer potential operators are stable on $L^2(\mathbb R^{n+1},\mathbb C)=L^2(\partial\mathbb R^{n+2}_+,\mathbb C)$ under complex, $L^\infty$ perturbations of the coefficient matrix. Subsequently, using this general result, we establish solvability of the Dirichlet, Neumann and Regularity problems for $\partial_t+\mathcal{L}$, by way of layer potentials and with data in $L^2$, assuming that the coefficient matrix is a small complex perturbation of either a constant matrix or of a real and symmetric matrix.

math.AP↗

arXiv2015

The weak-$A_\infty$ property of harmonic and $p$-harmonic measures implies uniform rectifiability

Let $E\subset \mathbb{R}^{n+1}$, $n\ge 2$, be an Ahlfors-David regular set of dimension $n$. We show that the weak-$A_\infty$ property of harmonic measure, for the open set $Ω:= \mathbb{R}^{n+1}\setminus E$, implies uniform rectifiability of $E$. More generally, we establish a similar result for the Riesz measure, $p$-harmonic measure, associated to the $p$-Laplace operator, $1<p<\infty$.

math.CA↗

arXiv2015

Boundary behavior of solutions to the parabolic p-Laplace equation

We establish boundary estimates for non-negative solutions to the p-parabolic equation in the degenerate range $p>2$. Our main results include new parabolic intrinsic Harnack chains in cylindrical NTA-domains together with sharp boundary decay estimates. If the underlying domain is $C^{1,1}$-regular, we establish a relatively complete theory of the boundary behavior, including boundary Harnack principles and Hölder continuity of the ratios of two solutions, as well as fine properties of associated boundary measures. There is an intrinsic waiting time phenomena present which plays a fundamental role throughout the paper. In particular, conditions on these waiting times rule out well-known examples of explicit solutions violating the boundary Harnack principle.

math.AP↗

arXiv2015

Reflected BSDE of Wiener-Poisson type in Time-dependent Domains

In this paper we study multi-dimensional reflected backward stochastic differential equations driven by Wiener-Poisson type processes. We prove existence and uniqueness of solutions, with reflection in the inward spatial normal direction, in the setting of certain time-dependent domains.

math.PR↗

arXiv2014

Tug-of-war, market manipulation and option pricing

We develop an option pricing model based on a tug-of-war game. This two-player zero-sum stochastic differential game is formulated in the context of a multi-dimensional financial market. The issuer and the holder try to manipulate asset price processes in order to minimize and maximize the expected discounted reward. We prove that the game has a value and that the value function is the unique viscosity solution to a terminal value problem for a parabolic partial differential equation involving the non-linear and completely degenerate infinity Laplace operator.

math.AP↗