arXiv subjects

Anton Schiela

Publications and source records attributed to Anton Schiela.

17 recordsLinked to original sources

arXiv2026

Stabilize-then-optimize: Feedback transformations as preconditioners in optimal control

Many numerical algorithms for optimal control leverage an elimination of the state via the control-to-state map such as condensed approaches or preconditioned conjugate gradient methods for the optimality system. As such, the norm of the control-to-state map directly enters the convergence estimates for these methods, e.g., via the condition number of the associated linear system. In this work we show that using feedback transformations one may reformulate the optimal control problem to decrease the norm of the (feedbacked) control-to-state map, leading to a drastic improvement of the involved condition numbers. We illustrate the abstract approach for ordinary and partial differential equations such as parabolic, hyperbolic or elliptic equations. For each of these problem classes we provide a constructive method to improve solution operator norms via feedbacks. Further, we showcase the efficacy of the method by means of various numerical examples with elliptic, parabolic and hyperbolic partial differential equations.

math.OC↗

arXiv2025

Newton's method for nonlinear mappings into vector bundles

We consider Newton's method for finding zeros of mappings from a manifold $\mathcal X$ into a vector bundle $\mathcal E$. In this setting a connection on $\mathcal E$ is required to render the Newton equation well defined, and a retraction on $\mathcal X$ is needed to compute a Newton update. We discuss local convergence in terms of suitable differentiability concepts, using a Banach space variant of a Riemannian distance. We also carry over an affine covariant damping strategy to our setting. Finally, we will illustrate our results by applying them to generalized non-symmetric eigenvalue problems and providing a numerical example.

math.DG↗

arXiv2025

Newton's method for nonlinear mappings into vector bundles Part II: Application to variational problems

We consider the solution of variational equations on manifolds by Newton's method. These problems can be expressed as root finding problems for mappings from infinite dimensional manifolds into dual vector bundles. We derive the differential geometric tools needed for the realization of Newton's method, equipped with an affine covariant damping strategy. We apply Newton's method to a couple of variational problems and show numerical results.

math.NA↗

arXiv2025

SensLI: Sensitivity-Based Layer Insertion for Neural Networks

The training of neural networks requires tedious and often manual tuning of the network architecture. We propose a systematic approach to inserting new layers during the training process. Our method eliminates the need to choose a fixed network size before training, is numerically inexpensive to execute and applicable to various architectures including fully connected feedforward networks, ResNets and CNNs. Our technique borrows ideas from constrained optimization and is based on first-order sensitivity information of the loss function with respect to the virtual parameters that additional layers, if inserted, would offer. In numerical experiments, our proposed sensitivity-based layer insertion technique (SensLI) exhibits improved performance on training loss and test error, compared to training on a fixed architecture, and reduced computational effort in comparison to training the extended architecture from the beginning. Our code is available on https://github.com/mathemml/SensLI.

cs.LG↗

arXiv2025

An Exponential Averaging Process with Strong Convergence Properties

Averaging, or smoothing, is a fundamental approach to obtain stable, de-noised estimates from noisy observations. In certain scenarios, observations made along trajectories of random dynamical systems are of particular interest. One popular smoothing technique for such a scenario is exponential moving averaging (EMA), which assigns observations a weight that decreases exponentially in their age, thus giving younger observations a larger weight. However, EMA fails to enjoy strong stochastic convergence properties, which stems from the fact that the weight assigned to the youngest observation is constant over time, preventing the noise in the averaged quantity from decreasing to zero. In this work, we consider an adaptation to EMA, which we call $p$-EMA, where the weights assigned to the last observations decrease to zero at a subharmonic rate. We provide stochastic convergence guarantees for this kind of averaging under mild assumptions on the autocorrelations of the underlying random dynamical system. We further discuss the implications of our results for a recently introduced adaptive step size control for Stochastic Gradient Descent (SGD), which uses $p$-EMA for averaging noisy observations.

stat.ML↗

arXiv2024

Adaptive Step Sizes for Preconditioned Stochastic Gradient Descent

This paper proposes a novel approach to adaptive step sizes in stochastic gradient descent (SGD) by utilizing quantities that we have identified as numerically traceable -- the Lipschitz constant for gradients and a concept of the local variance in search directions. Our findings yield a nearly hyperparameter-free algorithm for stochastic optimization, which has provable convergence properties and exhibits truly problem adaptive behavior on classical image classification tasks. Our framework is set in a general Hilbert space and thus enables the potential inclusion of a preconditioner through the choice of the inner product.

math.OC↗

arXiv2024

Nonlinear Fenchel Conjugates

The classical concept of Fenchel conjugation is tailored to extended real-valued functions defined on linear spaces. In this paper we generalize this concept to functions defined on arbitrary sets that do not necessarily bear any structure at all. This generalization is obtained by replacing linear test functions by general nonlinear ones. Thus, we refer to it as nonlinear Fenchel conjugation. We investigate elementary properties including the Fenchel-Moreau biconjugation theorem. Whenever the domain exhibits additional structure, the restriction to a suitable subset of test functions allows further results to be derived. For example, on smooth manifolds, the restriction to smooth test functions allows us to state the Fenchel-Young theorem for the viscosity Fréchet subdifferential. On Lie groups, the restriction to real-valued group homomorphisms relates nonlinear Fenchel conjugation to infimal convolution and yields a notion of convexity.

math.FA↗

arXiv2024

$L^\infty$-error bounds for approximations of the Koopman operator by kernel extended dynamic mode decomposition

Extended dynamic mode decomposition (EDMD) is a well-established method to generate a data-driven approximation of the Koopman operator for analysis and prediction of nonlinear dynamical systems. Recently, kernel EDMD (kEDMD) has gained popularity due to its ability to resolve the challenging task of choosing a suitable dictionary by using the kernel's canonical features and, thus, data-informed observables. In this paper, we provide the first pointwise bounds on the approximation error of kEDMD. The main idea consists of two steps. First, we show that the reproducing kernel Hilbert spaces of Wendland functions are invariant under the Koopman operator. Second, exploiting that the learning problem given by regression in the native norm can be recast as an interpolation problem, we prove our novel error bounds by using interpolation estimates. Finally, we validate our findings with numerical experiments.

math.DS↗

arXiv2023

Frobenius-Type Norms and Inner Products of Matrices and Linear Maps with Applications to Neural Network Training

The Frobenius norm is a frequent choice of norm for matrices. In particular, the underlying Frobenius inner product is typically used to evaluate the gradient of an objective with respect to matrix variable, such as those occuring in the training of neural networks. We provide a broader view on the Frobenius norm and inner product for linear maps or matrices, and establish their dependence on inner products in the domain and co-domain spaces. This shows that the classical Frobenius norm is merely one special element of a family of more general Frobenius-type norms. The significant extra freedom furnished by this realization can be used, among other things, to precondition neural network training.

cs.LG↗

arXiv2022

Inexact Proximal Newton methods in Hilbert spaces

We consider Proximal Newton methods with an inexact computation of update steps. To this end, we introduce two inexactness criteria which characterize sufficient accuracy of these update step and with the aid of these investigate global convergence and local acceleration of our method. The inexactness criteria are designed to be adequate for the Hilbert space framework we find ourselves in while traditional inexactness criteria from smooth Newton or finite dimensional Proximal Newton methods appear to be inefficient in this scenario. The performance of the method and its gain in effectiveness in contrast to the exact case are showcased considering a simple model problem in function space.

math.OC↗

arXiv2022

Efficient Model Predictive Control for Parabolic PDEs with Goal Oriented Error Estimation

We show how a posteriori goal oriented error estimation can be used to efficiently solve the subproblems occurring in a Model Predictive Control (MPC) algorithm. In MPC, only an initial part of a computed solution is implemented as a feedback, which motivates grid refinement particularly tailored to this context. To this end, we present a truncated cost functional as objective for goal oriented adaptivity and prove under stabilizability assumptions that error indicators decay exponentially outside the support of this quantity. This leads to very efficient time and space discretizations for MPC, which we will illustrate by means of various numerical examples.

math.OC↗

arXiv2022

Abstract nonlinear sensitivity and turnpike analysis and an application to semilinear parabolic PDEs

We analyze the sensitivity of the extremal equations that arise from the first order necessary optimality conditions of nonlinear optimal control problems with respect to perturbations of the dynamics and of the initial data. To this end, we present an abstract implicit function approach with scaled spaces. We will apply this abstract approach to problems governed by semilinear PDEs. In that context, we prove an exponential turnpike result and show that perturbations of the extremal equation's dynamics, e.g., discretization errors decay exponentially in time. The latter can be used for very efficient discretization schemes in a Model Predictive Controller, where only a part of the solution needs to be computed accurately. We showcase the theoretical results by means of two examples with a nonlinear heat equation on a two-dimensional domain.

math.OC↗

arXiv2021

Second order semi-smooth Proximal Newton methods in Hilbert spaces

We develop a globalized Proximal Newton method for composite and possibly non-convex minimization problems in Hilbert spaces. Additionally, we impose less restrictive assumptions on the composite objective functional considering differentiability and convexity than in existing theory. As far as differentiability of the smooth part of the objective function is concerned, we introduce the notion of second order semi-smoothness and discuss why it constitutes an adequate framework for our Proximal Newton method. However, both global convergence as well as local acceleration still pertain to hold in our scenario. Eventually, the convergence properties of our algorithm are displayed by solving a toy model problem in function space.

math.OC↗

arXiv2020

A primal dual projection algorithm for efficient constraint preconditioning

We consider a linear iterative solver for large scale linearly constrained quadratic minimization problems that arise, for example, in optimization with PDEs. By a primal-dual projection (PDP) iteration, which can be interpreted and analysed as a gradient method on a quotient space, the given problem can be solved by computing sulutions for a sequence of constrained surrogate problems, projections onto the feasible subspaces, and Lagrange multiplier updates. As a major application we consider a class of optimization problems with PDEs, where PDP can be applied together with a projected cg method using a block triangular constraint preconditioner. Numerical experiments show reliable and competitive performance for an optimal control problem in elasticity.

math.OC↗

arXiv2020

Fenchel Duality Theory and A Primal-Dual Algorithm on Riemannian Manifolds

This paper introduces a new notion of a Fenchel conjugate, which generalizes the classical Fenchel conjugation to functions defined on Riemannian manifolds. We investigate its properties, e.g.,~the Fenchel--Young inequality and the characterization of the convex subdifferential using the analogue of the Fenchel--Moreau Theorem. These properties of the Fenchel conjugate are employed to derive a Riemannian primal-dual optimization algorithm, and to prove its convergence for the case of Hadamard manifolds under appropriate assumptions. Numerical results illustrate the performance of the algorithm, which competes with the recently derived Douglas--Rachford algorithm on manifolds of nonpositive curvature. Furthermore, we show numerically that our novel algorithm even converges on manifolds of positive curvature.

math.NA↗

arXiv2020

An SQP method for equality constrained optimization on manifolds

We extend the class of SQP methods for equality constrained optimization to the setting of differentiable manifolds. The use of retractions and stratifications allows us to pull back the involved mappings to linear spaces. We study local quadratic convergence to minimizers. In addition we present a composite step method for globalization based on cubic regularization of the objective function and affine covariant damped Newton method for feasibility. We show transition to fast local convergence of this scheme. We test our method on equilibrium problems in finite elasticity where the stable equilibrium position of an inextensible transversely isotropic elastic rod under dead load is sought.

math.OC↗

arXiv2017

Optimal control of elliptic equations with positive measures

Optimal control problems without control costs in general do not possess solutions due to the lack of coercivity. However, unilateral constraints together with the assumption of existence of strictly positive solutions of a pre-adjoint state equation, are sufficient to obtain existence of optimal solutions in the space of Radon measures. Optimality conditions for these generalized minimizers can be obtained using Fenchel duality, which requires a non-standard perturbation approach if the control-to-observation mapping is not continuous (e.g., for Neumann boundary control in three dimensions). Combining a conforming discretization of the measure space with a semismooth Newton method allows the numerical solution of the optimal control problem.

math.OC↗