arXiv subjects

Jeffrey Cornelis

Publications and source records attributed to Jeffrey Cornelis.

8 recordsLinked to original sources

arXiv2021

Sequential Projected Newton method for regularization of nonlinear least squares problems

We develop a computationally efficient algorithm for the automatic regularization of nonlinear inverse problems based on the discrepancy principle. We formulate the problem as an equality constrained optimization problem, where the constraint is given by a least squares data fidelity term and expresses the discrepancy principle. The objective function is a convex regularization function that incorporates some prior knowledge, such as the total variation regularization function. Using the Jacobian matrix of the nonlinear forward model, we consider a sequence of quadratically constrained optimization problems that can all be solved using the Projected Newton method. We show that the solution of such a quadratically constrained sub-problem results in a descent direction for an exact merit function. This merit function can then be used to describe a formal line-search method. We also formulate a slightly more heuristic approach that simplifies the algorithm and allows for an inexact solution of the sequence of sub-problems. We illustrate the behavior of the algorithm using a number of numerical experiments, with Talbot-Lau X-ray phase contrast imaging as the main application. The numerical experiments confirm that the quadratically constrained sub-problems need not be solved with high accuracy in early iterations to make sufficient progress towards the solution. In addition, we show that the proposed method is able to produce reconstructions of similar quality compared to other state-of-the-art approaches with a significant reduction in computational time.

math.NA↗

arXiv2021

Convergence analysis of a regularized inexact interior-point method for linear programming problems

Interior-point methods for linear programming problems require the repeated solution of a linear system of equations. Solving these linear systems is non-trivial due to the severe ill-conditioning of the matrices towards convergence. This issue can be alleviated by incorporating suitable regularization terms in the linear programming problem. Regularization also allows us to efficiently handle rank deficient constraint matrices. We provide a convergence analysis of a regularized inexact interior-point method. The term `inexact' refers to the fact that we do not need to compute the true solution of the linear system of equations, only an approximation thereof. The formulation of the algorithm is sufficiently general such that specialized linear algebra routines developed in other work on inexact interior-point methods can also be incorporated in our regularized framework. In this work, we exploit the inexactness by using a mixed-precision solver for the linear system of equations. More specifically, we perform a Cholesky factorization in IEEE single precision and use it as a preconditioner for the Conjugate Gradient method. Numerical experiments illustrate the benefits of this approach applied to linear programming problems with a dense constraint matrix.

math.OC↗

arXiv2021

Krylov-Simplex method that minimizes the residual in $\ell_1$-norm or $\ell_\infty$-norm

The paper presents two variants of a Krylov-Simplex iterative method that combines Krylov and simplex iterations to minimize the residual $r = b-Ax$. The first method minimizes $\|r\|_\infty$, i.e. maximum of the absolute residuals. The second minimizes $\|r\|_1$, and finds the solution with the least absolute residuals. Both methods search for an optimal solution $x_k$ in a Krylov subspace which results in a small linear programming problem. A specialized simplex algorithm solves this projected problem and finds the optimal linear combination of Krylov basis vectors to approximate the solution. The resulting simplex algorithm requires the solution of a series of small dense linear systems that only differ by rank-one updates. The $QR$ factorization of these matrices is updated each iteration. We demonstrate the effectiveness of the methods with numerical experiments.

math.NA↗

arXiv2020

Projected Newton method for noise constrained $\ell_p$ regularization

Choosing an appropriate regularization term is necessary to obtain a meaningful solution to an ill-posed linear inverse problem contaminated with measurement errors or noise. The $\ell_p$ norm covers a wide range of choices for the regularization term since its behavior critically depends on the choice of $p$ and since it can easily be combined with a suitable regularization matrix. We develop an efficient algorithm that simultaneously determines the regularization parameter and corresponding $\ell_p$ regularized solution such that the discrepancy principle is satisfied. We project the problem on a low-dimensional Generalized Krylov subspace and compute the Newton direction for this much smaller problem. We illustrate some interesting properties of the algorithm and compare its performance with other state-of-the-art approaches using a number of numerical experiments, with a special focus of the sparsity inducing $\ell_1$ norm and edge-preserving total variation regularization.

math.NA↗

arXiv2020

Projected Newton Method for noise constrained Tikhonov regularization

Tikhonov regularization is a popular approach to obtain a meaningful solution for ill-conditioned linear least squares problems. A relatively simple way of choosing a good regularization parameter is given by Morozov's discrepancy principle. However, most approaches require the solution of the Tikhonov problem for many different values of the regularization parameter, which is computationally demanding for large scale problems. We propose a new and efficient algorithm which simultaneously solves the Tikhonov problem and finds the corresponding regularization parameter such that the discrepancy principle is satisfied. We achieve this by formulating the problem as a nonlinear system of equations and solving this system using a line search method. We obtain a good search direction by projecting the problem onto a low dimensional Krylov subspace and computing the Newton direction for the projected problem. This projected Newton direction, which is significantly less computationally expensive to calculate than the true Newton direction, is then combined with a backtracking line search to obtain a globally convergent algorithm, which we refer to as the Projected Newton method. We prove convergence of the algorithm and illustrate the improved performance over current state-of-the-art solvers with some numerical experiments.

math.NA↗

arXiv2019

Numerically Stable Recurrence Relations for the Communication Hiding Pipelined Conjugate Gradient Method

Pipelined Krylov subspace methods (also referred to as communication-hiding methods) have been proposed in the literature as a scalable alternative to classic Krylov subspace algorithms for iteratively computing the solution to a large linear system in parallel. For symmetric and positive definite system matrices the pipelined Conjugate Gradient method outperforms its classic Conjugate Gradient counterpart on large scale distributed memory hardware by overlapping global communication with essential computations like the matrix-vector product, thus hiding global communication. A well-known drawback of the pipelining technique is the (possibly significant) loss of numerical stability. In this work a numerically stable variant of the pipelined Conjugate Gradient algorithm is presented that avoids the propagation of local rounding errors in the finite precision recurrence relations that construct the Krylov subspace basis. The multi-term recurrence relation for the basis vector is replaced by two-term recurrences, improving stability without increasing the overall computational cost of the algorithm. The proposed modification ensures that the pipelined Conjugate Gradient method is able to attain a highly accurate solution independently of the pipeline length. Numerical experiments demonstrate a combination of excellent parallel performance and improved maximal attainable accuracy for the new pipelined Conjugate Gradient algorithm. This work thus resolves one of the major practical restrictions for the useability of pipelined Krylov subspace methods.

math.NA↗

arXiv2019

Improving strong scaling of the Conjugate Gradient method for solving large linear systems using global reduction pipelining

This paper presents performance results comparing MPI-based implementations of the popular Conjugate Gradient (CG) method and several of its communication hiding (or 'pipelined') variants. Pipelined CG methods are designed to efficiently solve SPD linear systems on massively parallel distributed memory hardware, and typically display significantly improved strong scaling compared to classic CG. This increase in parallel performance is achieved by overlapping the global reduction phase (MPI_Iallreduce) required to compute the inner products in each iteration by (chiefly local) computational work such as the matrix-vector product as well as other global communication. This work includes a brief introduction to the deep pipelined CG method for readers that may be unfamiliar with the specifics of the method. A brief overview of implementation details provides the practical tools required for implementation of the algorithm. Subsequently, easily reproducible strong scaling results on the US Department of Energy (DoE) NERSC machine 'Cori' (Phase I - Haswell nodes) on up to 1024 nodes with 16 MPI ranks per node are presented using an implementation of p(l)-CG that is available in the open source PETSc library. Observations on the staggering and overlap of the asynchronous, non-blocking global communication phases with communication and computational kernels are drawn from the experiments.

cs.DC↗

arXiv2019

The Communication-Hiding Conjugate Gradient Method with Deep Pipelines

Krylov subspace methods are among the most efficient solvers for large scale linear algebra problems. Nevertheless, classic Krylov subspace algorithms do not scale well on massively parallel hardware due to synchronization bottlenecks. Communication-hiding pipelined Krylov subspace methods offer increased parallel scalability by overlapping the time-consuming global communication phase with computations such as spmvs, hence reducing the impact of the global synchronization and avoiding processor idling. One of the first published methods in this class is the pipelined Conjugate Gradient method (p-CG). However, on large numbers of processors the communication phase may take much longer than the computation of a single spmv. This work extends the pipelined CG method to deeper pipelines, denoted as p(l)-CG, which allows further scaling when the global communication phase is the dominant time-consuming factor. By overlapping the global all-to-all reduction phase in each CG iteration with the next l spmvs (deep pipelining), the method hides communication latency behind additional computational work. The p(l)-CG algorithm is derived from similar principles as the existing p(l)-GMRES method and by exploiting operator symmetry. The p(l)-CG method is also compared to other Krylov subspace methods, including the closely related classic CG and D-Lanczos methods and the pipelined CG method by Ghysels et al.. By analyzing the maximal accuracy attainable by the p(l)-CG method it is shown that the pipelining technique induces a trade-off between performance and numerical stability. A preconditioned version of the algorithm is also proposed and storage requirements and performance estimates are discussed. Experimental results demonstrate the possible performance gains and the attainable accuracy of deeper pipelined CG for solving large scale symmetric linear systems.

cs.DC↗