arXiv subjects

Angxiu Ni

Publications and source records attributed to Angxiu Ni.

23 records · Page 2Linked to original sources

arXiv2019

Linear Range in Gradient Descent

This paper defines linear range as the range of parameter perturbations which lead to approximately linear perturbations in the states of a network. We compute linear range from the difference between actual perturbations in states and the tangent solution. Linear range is a new criterion for estimating the effectivenss of gradients and thus having many possible applications. In particular, we propose that the optimal learning rate at the initial stages of training is such that parameter changes on all minibatches are within linear range. We demonstrate our algorithm on two shallow neural networks and a ResNet.

math.OC↗

arXiv2019

Adjoint sensitivity analysis on chaotic dynamical systems by Non-Intrusive Least Squares Adjoint Shadowing (NILSAS)

We develop the NILSAS algorithm, which performs adjoint sensitivity analysis of chaotic systems via computing the adjoint shadowing direction. NILSAS constrains its minimization to the adjoint unstable subspace, and can be implemented with little modification to existing adjoint solvers. The computational cost of NILSAS is independent of the number of parameters. We demonstrate NILSAS on the Lorenz 63 system and a weakly turbulent three-dimensional flow over a cylinder.

physics.comp-ph↗

arXiv2018

Adjoint shadowing directions in hyperbolic systems for sensitivity analysis

For hyperbolic diffeomorphisms, we define adjoint shadowing directions as a bounded inhomogeneous adjoint solution whose initial condition has zero component in the unstable adjoint direction. For hyperbolic flows, we define adjoint shadowing directions similarly, with the additional requirement that the average of its inner-product with the trajectory direction is zero. In both cases, we show unique existence of adjoint shadowing directions, and how they can be used for adjoint sensitivity analysis. Our work set a theoretical foundation for efficient adjoint sensitivity methods for long-time-averaged objectives such as NILSAS.

math.DS↗

arXiv2017

Least Squares Shadowing method for sensitivity analysis of differential equations

For a parameterized hyperbolic system $\frac{du}{dt}=f(u,s)$ the derivative of the ergodic average $\langle J \rangle = \lim_{T \to \infty}\frac{1}{T}\int_0^T J(u(t),s)$ to the parameter $s$ can be computed via the Least Squares Shadowing algorithm (LSS). We assume that the sytem is ergodic which means that $\langle J \rangle$ depends only on $s$ (not on the initial condition of the hyperbolic system). After discretizing this continuous system using a fixed timestep, the algorithm solves a constrained least squares problem and, from the solution to this problem, computes the desired derivative $\frac{d\langle J \rangle}{ds}$. The purpose of this paper is to prove that the value given by the LSS algorithm approaches the exact derivative when the discretization timestep goes to $0$ and the timespan used to formulate the least squares problem grows to infinity.

math.DS↗

arXiv2017

Simplified Least Squares Shadowing sensitivity analysis for chaotic ODEs and PDEs

This paper develops a variant of the Least Squares Shadowing (LSS) method, which has successfully computed the derivative for several chaotic ODEs and PDEs. The development in this paper aims to simplify Least Squares Shadowing method by improving how time dilation is treated. Instead of adding an explicit time dilation term as in the original method, the new variant uses windowing, which can be simpler to implement, especially for PDEs.

math.DS↗