arXiv subjects

Wim Vanroose

Publications and source records attributed to Wim Vanroose.

At least 19 recordsLinked to original sources

arXiv2023

ResQPASS: an algorithm for bounded variable linear least squares with asymptotic Krylov convergence

We present the Residual Quadratic Programming Active-Set Subspace (ResQPASS) method that solves large-scale linear least-squares problems with bound constraints on the variables. The problem is solved by creating a series of small problems of increasing size by projecting onto the basis of residuals. Each projected problem is solved by the active-set method for convex quadratic programming, warm-started with a working set and solution from the previous problem. The method coincides with conjugate gradients (CG) or, equivalently, LSQR when none of the constraints is active. When only a few constraints are active the method converges, after a few initial iterations, like CG and LSQR. An analysis links the convergence to an asymptotic Krylov subspace. We also present an efficient implementation where QR factorizations of the projected problems are updated over the inner iterations and Cholesky or Gram-Schmidt over the outer iterations.

math.NA↗

arXiv2022

Solving for the low-rank tensor components of a scattering wave function

Atomic and molecular breakup reactions, such as multiple-ionisation, are described by a driven Schrödinger equation. This equation is equivalent to a high-dimensional Helmholtz equation and it has solutions that are outgoing waves, emerging from the target. We show that these waves can be described by a low-rank approximation. For 2D problems this it a matrix product of two low-rank matrices, for 3D problems it is a low-rank tensor decomposition. We propose an iterative method that solves, in an alternating way, for these low-rank components of the scattered wave. We illustrate the method with examples in 2D and 3D.

math.NA↗

arXiv2021

Sequential Projected Newton method for regularization of nonlinear least squares problems

We develop a computationally efficient algorithm for the automatic regularization of nonlinear inverse problems based on the discrepancy principle. We formulate the problem as an equality constrained optimization problem, where the constraint is given by a least squares data fidelity term and expresses the discrepancy principle. The objective function is a convex regularization function that incorporates some prior knowledge, such as the total variation regularization function. Using the Jacobian matrix of the nonlinear forward model, we consider a sequence of quadratically constrained optimization problems that can all be solved using the Projected Newton method. We show that the solution of such a quadratically constrained sub-problem results in a descent direction for an exact merit function. This merit function can then be used to describe a formal line-search method. We also formulate a slightly more heuristic approach that simplifies the algorithm and allows for an inexact solution of the sequence of sub-problems. We illustrate the behavior of the algorithm using a number of numerical experiments, with Talbot-Lau X-ray phase contrast imaging as the main application. The numerical experiments confirm that the quadratically constrained sub-problems need not be solved with high accuracy in early iterations to make sufficient progress towards the solution. In addition, we show that the proposed method is able to produce reconstructions of similar quality compared to other state-of-the-art approaches with a significant reduction in computational time.

math.NA↗

arXiv2021

Convergence analysis of a regularized inexact interior-point method for linear programming problems

Interior-point methods for linear programming problems require the repeated solution of a linear system of equations. Solving these linear systems is non-trivial due to the severe ill-conditioning of the matrices towards convergence. This issue can be alleviated by incorporating suitable regularization terms in the linear programming problem. Regularization also allows us to efficiently handle rank deficient constraint matrices. We provide a convergence analysis of a regularized inexact interior-point method. The term `inexact' refers to the fact that we do not need to compute the true solution of the linear system of equations, only an approximation thereof. The formulation of the algorithm is sufficiently general such that specialized linear algebra routines developed in other work on inexact interior-point methods can also be incorporated in our regularized framework. In this work, we exploit the inexactness by using a mixed-precision solver for the linear system of equations. More specifically, we perform a Cholesky factorization in IEEE single precision and use it as a preconditioner for the Conjugate Gradient method. Numerical experiments illustrate the benefits of this approach applied to linear programming problems with a dense constraint matrix.

math.OC↗

arXiv2021

Krylov-Simplex method that minimizes the residual in $\ell_1$-norm or $\ell_\infty$-norm

The paper presents two variants of a Krylov-Simplex iterative method that combines Krylov and simplex iterations to minimize the residual $r = b-Ax$. The first method minimizes $\|r\|_\infty$, i.e. maximum of the absolute residuals. The second minimizes $\|r\|_1$, and finds the solution with the least absolute residuals. Both methods search for an optimal solution $x_k$ in a Krylov subspace which results in a small linear programming problem. A specialized simplex algorithm solves this projected problem and finds the optimal linear combination of Krylov basis vectors to approximate the solution. The resulting simplex algorithm requires the solution of a series of small dense linear systems that only differ by rank-one updates. The $QR$ factorization of these matrices is updated each iteration. We demonstrate the effectiveness of the methods with numerical experiments.

math.NA↗

arXiv2020

Projected Newton method for noise constrained $\ell_p$ regularization

Choosing an appropriate regularization term is necessary to obtain a meaningful solution to an ill-posed linear inverse problem contaminated with measurement errors or noise. The $\ell_p$ norm covers a wide range of choices for the regularization term since its behavior critically depends on the choice of $p$ and since it can easily be combined with a suitable regularization matrix. We develop an efficient algorithm that simultaneously determines the regularization parameter and corresponding $\ell_p$ regularized solution such that the discrepancy principle is satisfied. We project the problem on a low-dimensional Generalized Krylov subspace and compute the Newton direction for this much smaller problem. We illustrate some interesting properties of the algorithm and compare its performance with other state-of-the-art approaches using a number of numerical experiments, with a special focus of the sparsity inducing $\ell_1$ norm and edge-preserving total variation regularization.

math.NA↗

arXiv2020

Projected Newton Method for noise constrained Tikhonov regularization

Tikhonov regularization is a popular approach to obtain a meaningful solution for ill-conditioned linear least squares problems. A relatively simple way of choosing a good regularization parameter is given by Morozov's discrepancy principle. However, most approaches require the solution of the Tikhonov problem for many different values of the regularization parameter, which is computationally demanding for large scale problems. We propose a new and efficient algorithm which simultaneously solves the Tikhonov problem and finds the corresponding regularization parameter such that the discrepancy principle is satisfied. We achieve this by formulating the problem as a nonlinear system of equations and solving this system using a line search method. We obtain a good search direction by projecting the problem onto a low dimensional Krylov subspace and computing the Newton direction for the projected problem. This projected Newton direction, which is significantly less computationally expensive to calculate than the true Newton direction, is then combined with a backtracking line search to obtain a globally convergent algorithm, which we refer to as the Projected Newton method. We prove convergence of the algorithm and illustrate the improved performance over current state-of-the-art solvers with some numerical experiments.

math.NA↗

arXiv2019

Improving strong scaling of the Conjugate Gradient method for solving large linear systems using global reduction pipelining

This paper presents performance results comparing MPI-based implementations of the popular Conjugate Gradient (CG) method and several of its communication hiding (or 'pipelined') variants. Pipelined CG methods are designed to efficiently solve SPD linear systems on massively parallel distributed memory hardware, and typically display significantly improved strong scaling compared to classic CG. This increase in parallel performance is achieved by overlapping the global reduction phase (MPI_Iallreduce) required to compute the inner products in each iteration by (chiefly local) computational work such as the matrix-vector product as well as other global communication. This work includes a brief introduction to the deep pipelined CG method for readers that may be unfamiliar with the specifics of the method. A brief overview of implementation details provides the practical tools required for implementation of the algorithm. Subsequently, easily reproducible strong scaling results on the US Department of Energy (DoE) NERSC machine 'Cori' (Phase I - Haswell nodes) on up to 1024 nodes with 16 MPI ranks per node are presented using an implementation of p(l)-CG that is available in the open source PETSc library. Observations on the staggering and overlap of the asynchronous, non-blocking global communication phases with communication and computational kernels are drawn from the experiments.

cs.DC↗

arXiv2019

Algorithm for the reconstruction of dynamic objects in CT-scanning using optical flow

Computed Tomography is a powerful imaging technique that allows non-destructive visualization of the interior of physical objects in different scientific areas. In traditional reconstruction techniques the object of interest is mostly considered to be static, which gives artefacts if the object is moving during the data acquisition. In this paper we present a method that, given only scan results of multiple successive scans, can estimate the motion and correct the CT-images for this motion assuming that the motion field is smooth over the complete domain using optical flow. The proposed method is validated on simulated scan data. The main contribution is that we show we can use the optical flow technique from imaging to correct CT-scan images for motion.

eess.IV↗

arXiv2019

Automatic exploration techniques for the numerical bifurcation study of the Ginzburg-Landau equation

This paper considers the extreme type-II Ginzburg-Landau equations, a nonlinear PDE model that describes the states of a wide range of superconductors. For two-dimensional grids, a robust method is developed that performs a numerical continuation of the equations, automatically exploring the whole solution landscape. The strength of the applied magnetic field is used as the bifurcation parameter. Our branch switching algorithm is based on Lyapunov-Schmidt reduction, but we will show that for an important class of grids an alternative method based on the equivariant branching lemma can be applied as well. The complete algorithm has been implemented in Python and tested for multiple examples. For each example a complete solution landscape was constructed, showing the robustness of the algorithm.

math.DS↗

arXiv2019

Numerically Stable Recurrence Relations for the Communication Hiding Pipelined Conjugate Gradient Method

Pipelined Krylov subspace methods (also referred to as communication-hiding methods) have been proposed in the literature as a scalable alternative to classic Krylov subspace algorithms for iteratively computing the solution to a large linear system in parallel. For symmetric and positive definite system matrices the pipelined Conjugate Gradient method outperforms its classic Conjugate Gradient counterpart on large scale distributed memory hardware by overlapping global communication with essential computations like the matrix-vector product, thus hiding global communication. A well-known drawback of the pipelining technique is the (possibly significant) loss of numerical stability. In this work a numerically stable variant of the pipelined Conjugate Gradient algorithm is presented that avoids the propagation of local rounding errors in the finite precision recurrence relations that construct the Krylov subspace basis. The multi-term recurrence relation for the basis vector is replaced by two-term recurrences, improving stability without increasing the overall computational cost of the algorithm. The proposed modification ensures that the pipelined Conjugate Gradient method is able to attain a highly accurate solution independently of the pipeline length. Numerical experiments demonstrate a combination of excellent parallel performance and improved maximal attainable accuracy for the new pipelined Conjugate Gradient algorithm. This work thus resolves one of the major practical restrictions for the useability of pipelined Krylov subspace methods.

math.NA↗

arXiv2019

The Communication-Hiding Conjugate Gradient Method with Deep Pipelines

Krylov subspace methods are among the most efficient solvers for large scale linear algebra problems. Nevertheless, classic Krylov subspace algorithms do not scale well on massively parallel hardware due to synchronization bottlenecks. Communication-hiding pipelined Krylov subspace methods offer increased parallel scalability by overlapping the time-consuming global communication phase with computations such as spmvs, hence reducing the impact of the global synchronization and avoiding processor idling. One of the first published methods in this class is the pipelined Conjugate Gradient method (p-CG). However, on large numbers of processors the communication phase may take much longer than the computation of a single spmv. This work extends the pipelined CG method to deeper pipelines, denoted as p(l)-CG, which allows further scaling when the global communication phase is the dominant time-consuming factor. By overlapping the global all-to-all reduction phase in each CG iteration with the next l spmvs (deep pipelining), the method hides communication latency behind additional computational work. The p(l)-CG algorithm is derived from similar principles as the existing p(l)-GMRES method and by exploiting operator symmetry. The p(l)-CG method is also compared to other Krylov subspace methods, including the closely related classic CG and D-Lanczos methods and the pipelined CG method by Ghysels et al.. By analyzing the maximal accuracy attainable by the p(l)-CG method it is shown that the pipelining technique induces a trade-off between performance and numerical stability. A preconditioned version of the algorithm is also proposed and storage requirements and performance estimates are discussed. Experimental results demonstrate the possible performance gains and the attainable accuracy of deeper pipelined CG for solving large scale symmetric linear systems.

cs.DC↗

arXiv2018

Projected Newton method for a system of Tikhonov-Morozov equations

In this paper we derive a Newton type method to solve the non-linear system formed by combining the Tikhonov normal equations and Morozov's discrepancy principle. We prove that by placing a bound on the step size of the Newton iterations the method will always converge to the solution. By projecting the problem onto a low dimensional Krylov subspace and using the method to solve the projected non-linear system we show that we can reduce the computational cost of the method.

math.NA↗

arXiv2018

Regula falsi based automatic regularization method for PDE constrained optimization

Many inverse problems can be described by a PDE model with unknown parameters that need to be calibrated based on measurements related to its solution. This can be seen as a constrained minimization problem where one wishes to minimize the mismatch between the observed data and the model predictions, including an extra regularization term, and use the PDE as a constraint. Often, a suitable regularization parameter is determined by solving the problem for a whole range of parameters -- e.g. using the L-curve -- which is computationally very expensive. In this paper we derive two methods that simultaneously solve the inverse problem and determine a suitable value for the regularization parameter. The first one is a direct generalization of the Generalized Arnoldi Tikhonov method for linear inverse problems. The second method is a novel method based on similar ideas, but with a number of advantages for nonlinear problems.

math.NA↗

arXiv2017

Fast derivatives of likelihood functionals for ODE based models using adjoint-state method

We consider time series data modeled by ordinary differential equations (ODEs), widespread models in physics, chemistry, biology and science in general. The sensitivity analysis of such dynamical systems usually requires calculation of various derivatives with respect to the model parameters. We employ the adjoint state method (ASM) for efficient computation of the first and the second derivatives of likelihood functionals constrained by ODEs with respect to the parameters of the underlying ODE model. Essentially, the gradient can be computed with a cost (measured by model evaluations) that is independent of the number of the ODE model parameters and the Hessian with a linear cost in the number of the parameters instead of the quadratic one. The sensitivity analysis becomes feasible even if the parametric space is high-dimensional. The main contributions are derivation and rigorous analysis of the ASM in the statistical context, when the discrete data are coupled with the continuous ODE model. Further, we present a highly optimized implementation of the results and its benchmarks on a number of problems. The results are directly applicable in (e.g.) maximum-likelihood estimation or Bayesian sampling of ODE based statistical models, allowing for faster, more stable estimation of parameters of the underlying ODE model.

stat.ME↗

arXiv2017

Numerically Stable Variants of the Communication-hiding Pipelined Conjugate Gradients Algorithm for the Parallel Solution of Large Scale Symmetric Linear Systems

By reducing the number of global synchronization bottlenecks per iteration and hiding communication behind useful computational work, pipelined Krylov subspace methods achieve significantly improved parallel scalability on present-day HPC hardware. However, this typically comes at the cost of a reduced maximal attainable accuracy. This paper presents and compares several stabilized versions of the communication-hiding pipelined Conjugate Gradients method. The main novel contribution of this work is the reformulation of the multi-term recurrence pipelined CG algorithm by introducing shifts in the recursions for specific auxiliary variables. These shifts reduce the amplification of local rounding errors on the residual. The stability analysis presented in this work provides a rigorous method for selection of the optimal shift value in practice. It is shown that, given a proper choice for the shift parameter, the resulting shifted pipelined CG algorithm restores the attainable accuracy and displays nearly identical robustness to local rounding error propagation compared to classical CG. Numerical results on a variety of SPD benchmark problems compare different stabilization techniques for the pipelined CG algorithm, showing that the shifted pipelined CG algorithm is able to attain a high accuracy while displaying excellent parallel performance.

math.NA↗

arXiv2017

The communication-hiding pipelined BiCGStab method for the parallel solution of large unsymmetric linear systems

A High Performance Computing alternative to traditional Krylov subspace methods, pipelined Krylov subspace solvers offer better scalability in the strong scaling limit compared to standard Krylov subspace methods for large and sparse linear systems. The typical synchronization bottleneck is mitigated by overlapping time-consuming global communication phases with local computations in the algorithm. This paper describes a general framework for deriving the pipelined variant of any Krylov subspace algorithm. The proposed framework was implicitly used to derive the pipelined Conjugate Gradient (p-CG) method in "Hiding global synchronization latency in the preconditioned Conjugate Gradient algorithm" by P. Ghysels and W. Vanroose, Parallel Computing, 40(7):224--238, 2014. The pipelining framework is subsequently illustrated by formulating a pipelined version of the BiCGStab method for the solution of large unsymmetric linear systems on parallel hardware. A residual replacement strategy is proposed to account for the possible loss of attainable accuracy and robustness by the pipelined BiCGStab method. It is shown that the pipelined algorithm improves scalability on distributed memory machines, leading to significant speedups compared to standard preconditioned BiCGStab.

cs.DC↗

arXiv2016

Analyzing the effect of local rounding error propagation on the maximal attainable accuracy of the pipelined Conjugate Gradient method

Pipelined Krylov subspace methods typically offer improved strong scaling on parallel HPC hardware compared to standard Krylov subspace methods for large and sparse linear systems. In pipelined methods the traditional synchronization bottleneck is mitigated by overlapping time-consuming global communications with useful computations. However, to achieve this communication hiding strategy, pipelined methods introduce additional recurrence relations for a number of auxiliary variables that are required to update the approximate solution. This paper aims at studying the influence of local rounding errors that are introduced by the additional recurrences in the pipelined Conjugate Gradient method. Specifically, we analyze the impact of local round-off effects on the attainable accuracy of the pipelined CG algorithm and compare to the traditional CG method. Furthermore, we estimate the gap between the true residual and the recursively computed residual used in the algorithm. Based on this estimate we suggest an automated residual replacement strategy to reduce the loss of attainable accuracy on the final iterative solution. The resulting pipelined CG method with residual replacement improves the maximal attainable accuracy of pipelined CG, while maintaining the efficient parallel performance of the pipelined method. This conclusion is substantiated by numerical results for a variety of benchmark problems.

math.NA↗