arXiv subjects

Evelyn Herberg

Publications and source records attributed to Evelyn Herberg.

8 recordsLinked to original sources

arXiv2026

A DC-Reformulation for Gradient-$L^0$-Constrained Problems

Cardinality constraints in optimization are commonly of $L^0$-type, and they lead to sparsely supported optimizers. An efficient way of dealing with these constraints algorithmically, when the objective functional is convex, is reformulating the constraint using the difference of suitable $L^1$- and largest-$K$-norms and subsequently solving a sequence of penalized subproblems in the difference-of-convex (DC) class. We extend this DC-reformulation approach to problems with $L^0$-type cardinality constraints on the support of the gradients, \ie, problems where sparsity of the gradient and thus piecewise constant solutions are the target. We apply the resulting algorithm to a classical energy minimization problem, an optimal control problem, and an image reconstruction problem.

math.OC↗

arXiv2025

SensLI: Sensitivity-Based Layer Insertion for Neural Networks

The training of neural networks requires tedious and often manual tuning of the network architecture. We propose a systematic approach to inserting new layers during the training process. Our method eliminates the need to choose a fixed network size before training, is numerically inexpensive to execute and applicable to various architectures including fully connected feedforward networks, ResNets and CNNs. Our technique borrows ideas from constrained optimization and is based on first-order sensitivity information of the loss function with respect to the virtual parameters that additional layers, if inserted, would offer. In numerical experiments, our proposed sensitivity-based layer insertion technique (SensLI) exhibits improved performance on training loss and test error, compared to training on a fixed architecture, and reduced computational effort in comparison to training the extended architecture from the beginning. Our code is available on https://github.com/mathemml/SensLI.

cs.LG↗

arXiv2023

Time Regularization in Optimal Time Variable Learning

Recently, optimal time variable learning in deep neural networks (DNNs) was introduced in arXiv:2204.08528. In this manuscript we extend the concept by introducing a regularization term that directly relates to the time horizon in discrete dynamical systems. Furthermore, we propose an adaptive pruning approach for Residual Neural Networks (ResNets), which reduces network complexity without compromising expressiveness, while simultaneously decreasing training time. The results are illustrated by applying the proposed concepts to classification tasks on the well known MNIST and Fashion MNIST data sets. Our PyTorch code is available on https://github.com/frederikkoehne/time_variable_learning.

cs.LG↗

arXiv2023

Lecture Notes: Neural Network Architectures

These lecture notes provide an overview of Neural Network architectures from a mathematical point of view. Especially, Machine Learning with Neural Networks is seen as an optimization problem. Covered are an introduction to Neural Networks and the following architectures: Feedforward Neural Network, Convolutional Neural Network, ResNet, and Recurrent Neural Network.

cs.LG↗

arXiv2022

An Optimal Time Variable Learning Framework for Deep Neural Networks

Feature propagation in Deep Neural Networks (DNNs) can be associated to nonlinear discrete dynamical systems. The novelty, in this paper, lies in letting the discretization parameter (time step-size) vary from layer to layer, which needs to be learned, in an optimization framework. The proposed framework can be applied to any of the existing networks such as ResNet, DenseNet or Fractional-DNN. This framework is shown to help overcome the vanishing and exploding gradient issues. Stability of some of the existing continuous DNNs such as Fractional-DNN is also studied. The proposed approach is applied to an ill-posed 3D-Maxwell's equation.

math.OC↗

arXiv2022

Variational discretization of one-dimensional elliptic optimal control problems with BV functions based on the mixed formulation

We consider optimal control of an elliptic two-point boundary value problem governed by functions of bounded variation (BV). The cost functional is composed of a tracking term for the state and the BV-seminorm of the control. We use the mixed formulation for the state equation together with the variational discretization approach, where we use the classical lowest order Raviart-Thomas finite elements for the state equation. Consequently the variational discrete control is a piecewise constant function over the finite element grid. We prove error estimates for the variational discretization approach in combination with the mixed formulation of the state equation and confirm our analytical findings with numerical experiments.

math.OC↗

arXiv2020

Variational discretization approach applied to an optimal control problem with bounded measure controls

We consider a parabolic optimal control problem with an initial measure control. The cost functional consists of a tracking term corresponding to the observation of the state at final time. Instead of a regularization term in the cost functional, we follow an approach by E. Casas and K. Kunisch and consider a bound on the measure norm of the initial control. The variational discretization of the problem together with the optimality conditions induce maximal discrete sparsity of the initial control, i.e. Dirac measures in space. We present numerical experiments to illustrate our approach.

math.OC↗

arXiv2019

Maximal discrete sparsity in parabolic optimal control with measures

We consider variational discretization of a parabolic optimal control problem governed by space-time measure controls. For the state discretization we use a Petrov-Galerkin method employing piecewise constant states and piecewise linear and continuous test functions in time. For the space discretization we use piecewise linear and continuous functions. As a result the controls are composed of Dirac measures in space-time, centered at points on the discrete space-time grid. We prove that the optimal discrete states and controls converge strongly in $L^q$ and weakly-$*$ in $\mathcal{M}$, respectively, to their smooth counterparts, where $q \in (1,\min\{2,1+2/d\}]$ is the spatial dimension. Furthermore, we compare our approach to a approach by Casas, E. and Kunisch, K., where the corresponding control problem is discretized employing a discontinuous Galerkin method for the state discretization and where the discrete controls are piecewise constant in time and Dirac measures in space. Numerical experiments highlight the features of our discrete approach.

math.OC↗