arXiv subjects

Matthias J. Ehrhardt

Publications and source records attributed to Matthias J. Ehrhardt.

At least 19 recordsLinked to original sources

arXiv2026

Greedy Learning to Optimize with Convergence Guarantees

Learning to optimize (L2O) is an approach that leverages training data to accelerate the solution of optimization problems. Many approaches use unrolling to parametrize the update step and learn optimal parameters. This requires choosing an unroll length in advance and typically incurs memory costs that grow with the number of unrolled iterations. Moreover, learned optimizers often lack convergence guarantees. We propose a greedy L2O framework that learns iteration-dependent parameters sequentially, by minimizing the average objective value after one additional optimization step. This converts the finite-unroll training problem into a sequence of one-step learning problems, allowing training to proceed until a stopping criterion is met while maintaining constant device-memory requirements with respect to the number of training iterations. We apply this framework to a preconditioned variant of Polyak's heavy-ball method, using several parameterizations of the learned preconditioners, including a convolutional parameterization for imaging problems. For linear parameterizations, the greedy learning subproblem is convex whenever the training objectives are convex, and admits closed-form solutions for least-squares objectives. We prove convergence in the training set even when the preconditioners are not necessarily symmetric nor positive definite. Convergence on a class of unseen functions is also obtained under certain assumptions. We test our learned algorithms on two inverse problems, image deblurring and Computed Tomography, on which learned convolutional preconditioners demonstrate improved empirical performance over classical optimization algorithms such as Nesterov's Accelerated Gradient Method and the quasi-Newton method L-BFGS.

math.OC↗

arXiv2026

PET Rapid Image Reconstruction Challenge (PETRIC)

Introduction: We describe the foundation of PETRIC, an image reconstruction challenge to minimise the computational runtime of related algorithms for Positron Emission Tomography (PET). Purpose: Although several similar challenges are well-established in the field of medical imaging, there have been no prior challenges for PET image reconstruction. Methods: Participants are provided with open-source software for implementation of their reconstruction algorithm(s). We define the objective function and reconstruct "gold standard" reference images, and provide metrics for quantifying algorithmic performance. We also received and curated phantom datasets (acquired with different scanners, radionuclides, and phantom types), which we further split into training and evaluation datasets. The automated computational framework of the challenge is released as open-source software. Results: Four teams with nine algorithms in total participated in the challenge. Their contributions made use of various tools from optimisation theory including preconditioning, stochastic gradients, and artificial intelligence. While most of the submitted approaches appear very similar in nature, their specific implementation lead to a range of algorithmic performance. Conclusion: As the first challenge for PET image reconstruction, PETRIC's solid foundations allow researchers to reuse its framework for evaluating new and existing image reconstruction methods on new or existing datasets. Variant versions of the challenge have and will continue to be launched in the future.

physics.med-ph↗

arXiv2026

Neural Fields for Highly Accelerated 2D Cine Phase Contrast MRI

2D cine phase contrast (CPC) MRI provides quantitative information on blood velocity and flow within the human vasculature. However, data acquisition is time-consuming, motivating the reconstruction of the velocity field from undersampled measurements to reduce scan times. In this work, neural fields are proposed as a continuous spatiotemporal parametrization of complex-valued images, jointly modeling magnitude and phase across multiple echoes to enable velocity estimation, and leveraging their inductive bias for the reconstruction of the velocity data. Additionally, to compensate for the oversmoothing tendency observed in neural-field reconstructions under severe undersampling, a simple voxel-based postprocessing step is introduced. The method is validated numerically in Cartesian and radial k-space with both high and low temporal resolution data. This approach achieves accurate reconstructions at high acceleration factors, with low errors even at 32$\times$ and 64$\times$ undersampling for the high temporal resolution data, and 16$\times$ for the low temporal resolution data, and consistently outperforms classical locally low-rank regularized voxel-based methods in both flow estimates and anatomical depiction.

eess.IV↗

arXiv2026

Learning Regularization Functionals for Inverse Problems: A Comparative Study

In recent years, a variety of learned regularization frameworks for solving inverse problems in imaging have emerged. These offer flexible modeling together with mathematical insights. The proposed methods differ in their architectural design and training strategies, making direct comparison challenging due to non-modular implementations. We address this gap by collecting and unifying the available code into a common framework. This unified view allows us to systematically compare the approaches and highlight their strengths and limitations, providing valuable insights into their future potential. We also provide concise descriptions of each method, complemented by practical guidelines.

cs.LG↗

arXiv2025

Bilevel Learning via Inexact Stochastic Gradient Descent

Bilevel optimization is a central tool in machine learning for high-dimensional hyperparameter tuning. Its applications are vast; for instance, in imaging it can be used for learning data-adaptive regularizers and optimizing forward operators in variational regularization. These problems are large in many ways: a lot of data is usually available to train a large number of parameters, calling for stochastic gradient-based algorithms. However, exact gradients with respect to parameters (so-called hypergradients) are not available, and their precision is usually linearly related to computational cost. Hence, algorithms must solve the problem efficiently without unnecessary precision. The design of such methods is still not fully understood, especially regarding how accuracy requirements and step size schedules affect theoretical guarantees and practical performance. Existing approaches introduce stochasticity at both the upper level (e.g., in sampling or mini-batch estimates) and the lower level (e.g., in solving the inner problem) to improve generalization, but they typically fix the number of lower-level iterations, which conflicts with asymptotic convergence assumptions. In this work, we advance the theory of inexact stochastic bilevel optimization. We prove convergence and establish rates under decaying accuracy and step size schedules, showing that with optimal configurations convergence occurs at an $\mathcal{O}(k^{-1/4})$ rate in expectation. Experiments on image denoising and inpainting with convex ridge regularizers and input-convex networks confirm our analysis: decreasing step sizes improve stability, accuracy scheduling is more critical than step size strategy, and adaptive preconditioning (e.g., Adam) further boosts performance. These results bridge theory and practice, providing convergence guarantees and practical guidance for large-scale imaging problems.

math.OC↗

arXiv2025

Stable neural networks and connections to continuous dynamical systems

The existence of instabilities, for example in the form of adversarial examples, has given rise to a highly active area of research concerning itself with understanding and enhancing the stability of neural networks. We focus on a popular branch within this area which draws on connections to continuous dynamical systems and optimal control, giving a bird's eye view of this area. We identify and describe the fundamental concepts that underlie much of the existing work in this area. Following this, we go into more detail on a specific approach to designing stable neural networks, developing the theoretical background and giving a description of how these networks can be implemented. We provide code that implements the approach that can be adapted and extended by the reader. The code further includes a notebook with a fleshed-out toy example on adversarial robustness of image classification that can be run without heavy requirements on the reader's computer. We finish by discussing this toy example so that the reader can interactively follow along on their computer. This work will be included as a chapter of a book on scientific machine learning, which is currently under revision and aimed at students.

math.NA↗

arXiv2025

A primal-dual algorithm for image reconstruction with input-convex neural network regularizers

We address the optimization problem in a data-driven variational reconstruction framework, where the regularizer is parameterized by an input-convex neural network (ICNN). While gradient-based methods are commonly used to solve such problems, they struggle to effectively handle non-smooth problems which often leads to slow convergence. Moreover, the nested structure of the neural network complicates the application of standard non-smooth optimization techniques, such as proximal algorithms. To overcome these challenges, we reformulate the problem and eliminate the network's nested structure. By relating this reformulation to epigraphical projections of the activation functions, we transform the problem into a convex optimization problem that can be efficiently solved using a primal-dual algorithm. We also prove that this reformulation is equivalent to the original variational problem. Through experiments on several imaging tasks, we show that the proposed approach not only outperforms subgradient methods and even accelerated methods in the smooth setting, but also facilitates the training of the regularizer itself.

math.OC↗

arXiv2025

Efficient gradient-based methods for bilevel learning via recycling Krylov subspaces

Many optimization problems require hyperparameters, i.e., parameters that must be pre-specified in advance, such as regularization parameters and parametric regularizers in variational regularization methods for inverse problems, and dictionaries in compressed sensing. A data-driven approach to determine appropriate hyperparameter values is via a nested optimization framework known as bilevel learning. Even when it is possible to employ a gradient-based solver to the bilevel optimization problem, construction of the gradients, known as hypergradients, is computationally challenging, each one requiring both a solution of a minimization problem and a linear system solve. These systems do not change much during the iterations, which motivates us to apply recycling Krylov subspace methods, wherein information from one linear system solve is re-used to solve the next linear system. Existing recycling strategies often employ eigenvector approximations called Ritz vectors. In this work we propose a novel recycling strategy based on a new concept, Ritz generalized singular vectors, which acknowledge the bilevel setting. Additionally, while existing iterative methods primarily terminate according to the residual norm, this new concept allows us to define a new stopping criterion that directly approximates the error of the associated hypergradient. The proposed approach is validated through extensive numerical testing in the context of inverse problems in imaging.

math.OC↗

arXiv2025

An Adaptively Inexact Method for Bilevel Learning Using Primal-Dual Style Differentiation

We consider a bilevel learning framework for learning linear operators. In this framework, the learnable parameters are optimized via a loss function that also depends on the minimizer of a convex optimization problem (denoted lower-level problem). We utilize an iterative algorithm called `piggyback' to compute the gradient of the loss and minimizer of the lower-level problem. Given that the lower-level problem is solved numerically, the loss function and thus its gradient can only be computed inexactly. To estimate the accuracy of the computed hypergradient, we derive an a-posteriori error bound, which provides guides for setting the tolerance for the lower-level problem, as well as the piggyback algorithm. To efficiently solve the upper-level optimization, we also propose an adaptive method for choosing a suitable step-size. To illustrate the proposed method, we consider a few learned regularizer problems, such as training an input-convex neural network.

math.OC↗

arXiv2025

Fast PET Reconstruction with Variance Reduction and Prior-Aware Preconditioning

We investigate subset-based optimization methods for positron emission tomography (PET) image reconstruction incorporating a regularizing prior. PET reconstruction methods that use a prior, such as the relative difference prior (RDP), are of particular relevance, as they are widely used in clinical practice and have been shown to outperform conventional early-stopped and post-smoothed ordered subsets expectation maximization (OSEM). Our study evaluates these methods on both simulated data and real brain PET scans from the 2024 PET Rapid Image Reconstruction Challenge (PETRIC), where the main objective was to achieve RDP-regularized reconstructions as fast as possible, making it an ideal benchmark. Our key finding is that incorporating the effect of the prior into the preconditioner is crucial for ensuring fast and stable convergence. In extensive simulation experiments, we compare several stochastic algorithms -- including Stochastic Gradient Descent (SGD), Stochastic Averaged Gradient Amelioré (SAGA), and Stochastic Variance Reduced Gradient (SVRG) -- under various algorithmic design choices and evaluate their performance for varying count levels and regularization strengths. The results show that SVRG and SAGA outperformed SGD, with SVRG demonstrating a slight overall advantage. The insights gained from these simulations directly contributed to the design of our submitted algorithms, which formed the basis of the winning contribution to the PETRIC 2024 challenge.

physics.med-ph↗

arXiv2025

Complex extension of optical flow and its practical evaluation for undersampled dynamic MRI

Reconstructing high-quality images from undersampled dynamic MRI data is a challenging task and important for the success of this imaging modality. To remedy the naturally occurring artifacts due to measurement undersampling, one can incorporate a motion model into the reconstruction so that information can propagate across time frames. Current models for MRI imaging are using the optical flow equation. However, they are based on real-valued images. Here, we generalise the optical flow equation to complex-valued images and demonstrate, based on two real cardiac MRI datasets, that the new model is capable of improving image quality.

math.OC↗

arXiv2025

An adaptively inexact first-order method for bilevel optimization with application to hyperparameter learning

Various tasks in data science are modeled utilizing the variational regularization approach, where manually selecting regularization parameters presents a challenge. The difficulty gets exacerbated when employing regularizers involving a large number of hyperparameters. To overcome this challenge, bilevel learning can be employed to learn such parameters from data. However, neither exact function values nor exact gradients with respect to the hyperparameters are attainable, necessitating methods that only rely on inexact evaluation of such quantities. State-of-the-art inexact gradient-based methods a priori select a sequence of the required accuracies and cannot identify an appropriate step size since the Lipschitz constant of the hypergradient is unknown. In this work, we propose an algorithm with backtracking line search that only relies on inexact function evaluations and hypergradients and show convergence to a stationary point. Furthermore, the proposed algorithm determines the required accuracy dynamically rather than manually selected before running it. Our numerical experiments demonstrate the efficiency and feasibility of our approach for hyperparameter estimation on a range of relevant problems in imaging and data science such as total variation and field of experts denoising and multinomial logistic regression. Particularly, the results show that the algorithm is robust to its own hyperparameters such as the initial accuracies and step size.

math.OC↗

arXiv2025

Bilevel Learning with Inexact Stochastic Gradients

Bilevel learning has gained prominence in machine learning, inverse problems, and imaging applications, including hyperparameter optimization, learning data-adaptive regularizers, and optimizing forward operators. The large-scale nature of these problems has led to the development of inexact and computationally efficient methods. Existing adaptive methods predominantly rely on deterministic formulations, while stochastic approaches often adopt a doubly-stochastic framework with impractical variance assumptions, enforces a fixed number of lower-level iterations, and requires extensive tuning. In this work, we focus on bilevel learning with strongly convex lower-level problems and a nonconvex sum-of-functions in the upper-level. Stochasticity arises from data sampling in the upper-level which leads to inexact stochastic hypergradients. We establish their connection to state-of-the-art stochastic optimization theory for nonconvex objectives. Furthermore, we prove the convergence of inexact stochastic bilevel optimization under mild assumptions. Our empirical results highlight significant speed-ups and improved generalization in imaging tasks such as image denoising and deblurring in comparison with adaptive deterministic bilevel methods.

math.OC↗

arXiv2024

A Guide to Stochastic Optimisation for Large-Scale Inverse Problems

Stochastic optimisation algorithms are the de facto standard for machine learning with large amounts of data. Handling only a subset of available data in each optimisation step dramatically reduces the per-iteration computational costs, while still ensuring significant progress towards the solution. Driven by the need to solve large-scale optimisation problems as efficiently as possible, the last decade has witnessed an explosion of research in this area. Leveraging the parallels between machine learning and inverse problems has allowed harnessing the power of this research wave for solving inverse problems. In this survey, we provide a comprehensive account of the state-of-the-art in stochastic optimisation from the viewpoint of variational regularisation for inverse problems where the solution is modelled as minimising an objective function. We present algorithms with diverse modalities of problem randomisation and discuss the roles of variance reduction, acceleration, higher-order methods, and other algorithmic modifications, and compare theoretical results with practical behaviour. We focus on the potential and the challenges for stochastic optimisation that are unique to variational regularisation for inverse imaging problems and are not commonly encountered in machine learning. We conclude the survey with illustrative examples from imaging on linear inverse problems to examine the advantages and disadvantages that this new generation of algorithms bring to the field of inverse problems.

math.NA↗

arXiv2024

Stochastic Multiresolution Image Sketching for Inverse Imaging Problems

A challenge in high-dimensional inverse problems is developing iterative solvers to find the accurate solution of regularized optimization problems with low computational cost. An important example is computed tomography (CT) where both image and data sizes are large and therefore the forward model is costly to evaluate. Since several years algorithms from stochastic optimization are used for tomographic image reconstruction with great success by subsampling the data. Here we propose a novel way how stochastic optimization can be used to speed up image reconstruction by means of image domain sketching such that at each iteration an image of different resolution is being used. Hence, we coin this algorithm ImaSk. By considering an associated saddle-point problem, we can formulate ImaSk as a gradient-based algorithm where the gradient is approximated in the same spirit as the stochastic average gradient amélioré (SAGA) and uses at each iteration one of these multiresolution operators at random. We prove that ImaSk is linearly converging for linear forward models with strongly convex regularization functions. Numerical simulations on CT show that ImaSk is effective and increasing the number of multiresolution operators reduces the computational time to reach the modeled solution.

math.NA↗

arXiv2024

Analysis of Primal-Dual Langevin Algorithms

We analyze a recently proposed class of algorithms for the problem of sampling from probability distributions $μ^\ast$ in $\mathbb{R}^d$ with a Lebesgue density of the form $μ^\ast(x) \propto \exp(-f(Kx)-g(x))$, where $K$ is a linear operator and $f,g$ convex and non-smooth. The method is a generalization of the primal-dual hybrid gradient optimization algorithm to a sampling scheme. We give the iteration's continuous time limit, a stochastic differential equation in the joint primal-dual variable, and its mean field limit Fokker-Planck equation. Under mild conditions, the scheme converges to a unique stationary state in continuous and discrete time. Contrary to purely primal overdamped Langevin diffusion, the stationary state in continuous time does not have $μ^\ast$ as its primal marginal. Thus, further analysis is carried out to bound the bias induced by the partial dualization, and potentially correct for it in the diffusion. Time discretizations of the diffusion lead to implementable algorithms, but, as is typical in Langevin Monte Carlo methods, introduce further bias. We prove bounds for these discretization errors, which allow to give convergence results relating the produced samples to the target. We demonstrate our findings numerically first on small-scale examples in which we can exactly verify the theoretical results, and subsequently on typical examples of larger scale from Bayesian imaging inverse problems.

math.OC↗

arXiv2024

Accelerated Convergent Motion Compensated Image Reconstruction

Motion correction aims to prevent motion artefacts which may be caused by respiration, heartbeat, or head movements for example. In a preliminary step, the measured data is divided in gates corresponding to motion states, and displacement maps from a reference state to each motion state are estimated. One common technique to perform motion correction is the motion compensated image reconstruction framework, where the displacement maps are integrated into the forward model corresponding to gated data. For standard algorithms, the computational cost per iteration increases linearly with the number of gates. In order to accelerate the reconstruction, we propose the use of a randomized and convergent algorithm whose per iteration computational cost scales constantly with the number of gates. We show improvement on theoretical rates of convergence and observe the predicted speed-up on two synthetic datasets corresponding to rigid and non-rigid motion.

math.OC↗

arXiv2024

A geometric integration approach to smooth optimisation: Foundations of the discrete gradient method

Discrete gradient methods are geometric integration techniques that can preserve the dissipative structure of gradient flows. Due to the monotonic decay of the function values, they are well suited for general convex and nonconvex optimisation problems. Both zero- and first-order algorithms can be derived from the discrete gradient method by selecting different discrete gradients. In this paper, we present a thorough analysis of the discrete gradient method for optimisation which provides a solid theoretical foundation. We show that the discrete gradient method is well-posed by proving the existence of iterates for any positive time step, as well as uniqueness in some cases, and propose an efficient method for solving the associated discrete gradient equation. Moreover, we establish an $O(1/k)$ convergence rate for convex objectives and prove linear convergence if instead the Polyak-Lojasiewicz inequality is satisfied. The analysis is carried out for three discrete gradients-the Gonzalez discrete gradient, the mean value discrete gradient, and the Itoh-Abe discrete gradient, as well as for a randomised Itoh-Abe method. Our theoretical results are illustrated with a variety of numerical experiments, and we furthermore demonstrate that the methods are robust with respect to stiffness.

math.OC↗