Searcharxiv⌕ Search

arXiv subjects

Yuesheng Xu

Publications and source records attributed to Yuesheng Xu.

49 records · Page 3Linked to original sources

Convergence Rate Analysis for Fixed-Point Iterations of Generalized Averaged Nonexpansive Operators

We estimate convergence rates for fixed-point iterations of a class of nonlinear operators which are partially motivated from solving convex optimization problems. We introduce the notion of the generalized averaged nonexpansive (GAN) operator with a positive exponent, and provide a convergence rate analysis of the fixed-point iteration of the GAN operator. The proposed generalized averaged nonexpansiveness is weaker than the averaged nonexpansiveness while stronger than nonexpansiveness. We show that the fixed-point iteration of a GAN operator with a positive exponent converges to its fixed-point and estimate the local convergence rate (the convergence rate in terms of the distance between consecutive iterates) according to the range of the exponent. We prove that the fixed-point iteration of a GAN operator with a positive exponent strictly smaller than 1 can achieve an exponential global convergence rate (the convergence rate in terms of the distance between an iterate and the solution). Furthermore, we establish the global convergence rate of the fixed-point iteration of a GAN operator, depending on both the exponent of generalized averaged nonexpansiveness and the exponent of the H$\ddot{\text{o}}$lder regularity, if the GAN operator is also H$\ddot{\text{o}}$lder regular. We then apply the established theory to three types of convex optimization problems that appear often in data science to design fixed-point iterative algorithms for solving these optimization problems and to analyze their convergence properties.

math.OC↗

An $L^p$-weak Galerkin method for second order elliptic equations in non-divergence form

This article presents a new primal-dual weak Galerkin method for second order elliptic equations in non-divergence form. The new method is devised as a constrained $L^p$-optimization problem with constraints that mimic the second order elliptic equation by using the discrete weak Hessian locally on each element. An equivalent min-max characterization is derived to show the existence and uniqueness of the numerical solution. Optimal order error estimates are established for the numerical solution under the discrete $W^{2,p}$ norm, as well as the standard $W^{1,p}$ and $L^p$ norms. An equivalent characterization of the optimization problem in term of a system of fixed-point equations via the proximity operator is presented. An iterative algorithm is designed based on the fixed-point equations to solve the optimization problems. Implementation of the iterative algorithm is studied and convergence of the iterative algorithm is established. Numerical experiments for both smooth and non-smooth coefficients problems are presented to verify the theoretical findings.

math.NA↗

Representer Theorems in Banach Spaces: Minimum Norm Interpolation, Regularized Learning and Semi-Discrete Inverse Problems

Constructing or learning a function from a finite number of sampled data points (measurements) is a fundamental problem in science and engineering. This is often formulated as a minimum norm interpolation problem, regularized learning problem or, in general, a semi-discrete inverse problem, in certain functional spaces. The choice of an appropriate space is crucial for solutions of these problems. Motivated by sparse representations of the reconstructed functions such as compressed sensing and sparse learning, much of the recent research interest has been directed to considering these problems in certain Banach spaces in order to obtain their sparse solutions, which is a feasible approach to overcome challenges coming from the big data nature of most practical applications. It is the goal of this paper to provide a systematic study of the representer theorems for these problems in Banach spaces. There are a few existing results for these problems in a Banach space, with all of them regarding implicit representer theorems. We aim at obtaining explicit representer theorems based on which convenient solution methods will then be developed. For the minimum norm interpolation, the explicit representer theorems enable us to express the infimum in terms of the norm of the linear combination of the interpolation functionals. For the purpose of developing efficient computational algorithms, we establish the fixed-point equation formulation of solutions of these problems. We reveal that unlike in a Hilbert space, in general, solutions of these problems in a Banach space may not be able to be reduced to truly finite dimensional problems (with certain infinite dimensional components hidden). We demonstrate how this obstacle can be removed, reducing the original problem to a truly finite dimensional one, in the special case when the Banach space is $\ell_1(\mathbb{N})$.

math.FA↗

Multiplicative Noise Removal: Nonlocal Low-Rank Model and Its Proximal Alternating Reweighted Minimization Algorithm

The goal of this paper is to develop a novel numerical method for efficient multiplicative noise removal. The nonlocal self-similarity of natural images implies that the matrices formed by their nonlocal similar patches are low-rank. By exploiting this low-rank prior with application to multiplicative noise removal, we propose a nonlocal low-rank model for this task and develop a proximal alternating reweighted minimization (PARM) algorithm to solve the optimization problem resulting from the model. Specifically, we utilize a generalized nonconvex surrogate of the rank function to regularize the patch matrices and develop a new nonlocal low-rank model, which is a nonconvex nonsmooth optimization problem having a patchwise data fidelity and a generalized nonlocal low-rank regularization term. To solve this optimization problem, we propose the PARM algorithm, which has a proximal alternating scheme with a reweighted approximation of its subproblem. A theoretical analysis of the proposed PARM algorithm is conducted to guarantee its global convergence to a critical point. Numerical experiments demonstrate that the proposed method for multiplicative noise removal significantly outperforms existing methods such as the benchmark SAR-BM3D method in terms of the visual quality of the denoised images, and the PSNR (the peak-signal-to-noise ratio) and SSIM (the structural similarity index measure) values.

math.OC↗

Dynamic PET cardiac and parametric image reconstruction: a fixed-point proximity gradient approach using patch-based DCT and tensor SVD regularization

Our aim was to enhance visual quality and quantitative accuracy of dynamic positron emission tomography (PET)uptake images by improved image reconstruction, using sophisticated sparse penalty models that incorporate both 2D spatial+1D temporal (3DT) information. We developed two new 3DT PET reconstruction algorithms, incorporating different temporal and spatial penalties based on discrete cosine transform (DCT)w/ patches, and tensor nuclear norm (TNN) w/ patches, and compared to frame-by-frame methods; conventional 2D ordered subsets expectation maximization (OSEM) w/ post-filtering and 2D-DCT and 2D-TNN. A 3DT brain phantom with kinetic uptake (2-tissue model), and a moving 3DT cardiac/lung phantom was simulated and reconstructed. For the cardiac/lung phantom, an additional cardiac gated 2D-OSEM set was reconstructed. The structural similarity index (SSIM) and relative root mean squared error (rRMSE) relative ground truth was investigated. The image derived left ventricular (LV) volume for the cardiac/lung images was found by region growing and parametric images of the brain phantom were calculated. For the cardiac/lung phantom, 3DT-TNN yielded optimal images, and 3DT-DCT was best for the brain phantom. The optimal LV volume from the 3DT-TNN images was on average 11 and 55 percentage points closer to the true value compared to cardiac gated 2D-OSEM and 2D-OSEM respectively. Compared to 2D-OSEM, parametric images based on 3DT-DCT images generally had smaller bias and higher SSIM. Our novel methods that incorporate both 2D spatial and 1D temporal penalties produced dynamic PET images of higher quality than conventional 2D methods, w/o need for post-filtering. Breathing and cardiac motion were simultaneously captured w/o need for respiratory or cardiac gating. LV volumes were better recovered, and subsequently fitted parametric images were generally less biased and of higher quality.

cs.CV↗

Computing Integrals Involved the Gaussian Function with a Small Standard Deviation

We develop efficient numerical integration methods for computing an integral whose integrand is a product of a smooth function and the Gaussian function with a small standard deviation. Traditional numerical integration methods applied to the integral normally lead to poor accuracy due to the rapid change in high order derivatives of its integrand when the standard deviation is small. The proposed quadrature schemes are based on graded meshes designed according to the standard deviation so that the quadrature errors on the resulting subintervals are approximately equal. The integral in each subinterval is then computed by considering the Gaussian function as a weight function and interpolating the smooth factor of the integrand at the Chebyshev points of the first kind. For a finite order differentiable factor, we design a quadrature scheme having accuracy of a polynomial order and for an infinitely differentiable factor of the integrand, we design a quadrature scheme having accuracy of an exponential order. Numerical results are presented to confirm the accuracy of these proposed quadrature schemes.

math.NA↗

Generalized Mercer Kernels and Reproducing Kernel Banach Spaces

This article studies constructions of reproducing kernel Banach spaces (RKBSs) which may be viewed as a generalization of reproducing kernel Hilbert spaces (RKHSs). A key point is to endow Banach spaces with reproducing kernels such that machine learning in RKBSs can be well-posed and of easy implementation. First we verify many advanced properties of the general RKBSs such as density, continuity, separability, implicit representation, imbedding, compactness, representer theorem for learning methods, oracle inequality, and universal approximation. Then, we develop a new concept of generalized Mercer kernels to construct $p$-norm RKBSs for $1\leq p\leq\infty$. The $p$-norm RKBSs preserve the same simple format as the Mercer representation of RKHSs. Moreover, the $p$-norm RKBSs are isometrically equivalent to the standard $p$-norm spaces of countable sequences. Hence, the $p$-norm RKBSs possess more geometrical structures than RKHSs including sparsity. The generalized Mercer kernels also cover many well-known kernels, for example, min kernels, Gaussian kernels, and power series kernels. Finally, we propose to solve the support vector machines in the $p$-norm RKBSs, which are to minimize the regularized empirical risks over the $p$-norm RKBSs. We show that the infinite dimensional support vector machines in the $p$-norm RKBSs can be equivalently transferred to finite dimensional convex optimization problems such that we obtain the finite dimensional representations of the support vector machine solutions for practical applications. In particular, we verify that some special support vector machines in the $1$-norm RKBSs are equivalent to the classical $1$-norm sparse regressions. This gives fundamental supports of a novel learning tool called sparse learning methods to be investigated in our next research project.

math.FA↗

Two-step Fixed-point proximity algorithms for multi-block separable convex problems

Multi-block separable convex problems recently received considerable attention. This class of optimization problems minimizes a separable convex objective function with linear constraints. The algorithmic challenges come from the fact that the classic alternating direction method of multipliers (ADMM) for the problem is not necessarily convergent. However, it is observed that ADMM outperforms numerically many of its variants with guaranteed theoretical convergence. The goal of this paper is to develop convergent and computationally efficient algorithms for solving multi-block separable convex problems. We first characterize the solutions of the optimization problems by proximity operators of the convex functions involved in their objective function. We then design a two-step fixed-point iterative scheme for solving these problems based on the characterization. We further prove convergence of the iterative scheme and show that it has O(1/k) convergence rate in the ergodic sense and the sense of the partial primal-dual gap, where k denotes the iteration number. Moreover, we derive specific two-step fixed-point proximity algorithms (2SFPPA) from the proposed iterative scheme and establish their global convergence. Numerical experiments for solving the sparse MRI problem demonstrate the numerical efficiency of the proposed 2SFPPA.

math.OC↗

Operator Reproducing Kernel Hilbert Spaces

Motivated by the need of processing functional-valued data, or more general, operatorvalued data, we introduce the notion of the operator reproducing kernel Hilbert space (ORKHS). This space admits a unique operator reproducing kernel which reproduces a family of continuous linear operators on the space. The theory of ORKHSs and the associated operator reproducing kernels are established. A special class of ORKHSs, known as the perfect ORKHSs, are studied, which reproduce the family of the standard point-evaluation operators and at the same time another different family of continuous linear operators. The perfect ORKHSs are characterized in terms of features, especially for those with respect to integral operators. In particular, several specific examples of the perfect ORKHSs are presented. We apply the theory of ORKHSs to sampling and regularized learning, where operator-valued data are considered. Specifically, a general complete reconstruction formula from linear operators values is established in the framework of ORKHSs. The average sampling and the reconstruction of vector-valued functions are considered in specific ORKHSs. We also investigate in the ORKHSs setting the regularized learning schemes, which learn a target element from operator-valued data. The desired representer theorems of the learning problems are established to demonstrate the key roles played by the ORKHSs and the operator reproducing kernels in machine learning from operator-valued data. We finally point out that the continuity of linear operators, used to obtain the operator-valued data, on an ORKHS is necessary for the stability of the numerical reconstruction algorithm using the resulting data.

math.FA↗

Oscillation Preserving Galerkin Methods for Fredholm Integral Equations of the Second Kind with Oscillatory Kernels

Solutions of Fredholm integral equations of the second kind with oscillatory kernels likely exhibit oscillation. Standard numerical methods applied to solving equations of this type have poor numerical performance due to the influence of the highly rapid oscillation in the solutions. Understanding of the oscillation of the solutions is still inadequate in the literature and thus it requires further investigation. For this purpose, we introduce a notion to describe the degree of oscillation of an oscillatory function based on the dependence of its norm in a certain function space on the wavenumber. Based on this new notion, we construct structured oscillatory spaces with oscillatory structures. The structured spaces with a specific oscillatory structure can capture the oscillatory components of the solutions of Fredholm integral equations with oscillatory kernels. We then further propose oscillation preserving Galerkin methods for solving the equations by incorporating the standard approximation subspace of spline functions with a finite number of oscillatory functions which capture the oscillation of the exact solutions of the integral equations. We prove that the proposed methods have the optimal convergence order uniformly with respect to the wavenumber and they are numerically stable. A numerical example is presented to confirm the theoretical estimates.

math.NA↗

Computing Highly Oscillatory Integrals

We develop two classes of composite moment-free numerical quadratures for computing highly oscillatory integrals having integrable singularities and stationary points. The first class of the quadrature rules has a polynomial order of convergence and the second class has an exponential order of convergence. We first modify the moment-free Filon-type method for the oscillatory integrals without a singularity or a stationary point to accelerate their convergence. We then consider the oscillatory integrals without a singularity or a stationary point and then those with singularities and stationary points. The composite quadrature rules are developed based on partitioning the integration domain according to the wave number and the singularity of the integrand. The integral defined on a subinterval has either a weak singularity without rapid oscillation or oscillation without a singularity. The classical quadrature rules for weakly singular integrals using graded points are employed for the singular integral without rapid oscillation and the modified moment-free Filon-type method is used for the oscillatory integrals without a singularity. Unlike the existing methods, the proposed methods do not have to compute the inverse of the oscillator. Numerical experiments are presented to demonstrate the approximation accuracy and the computational efficiency of the proposed methods. Numerical results show that the proposed methods outperform methods published most recently.

math.NA↗

Efficient First Order Methods for Linear Composite Regularizers

A wide class of regularization problems in machine learning and statistics employ a regularization term which is obtained by composing a simple convex function ωwith a linear transformation. This setting includes Group Lasso methods, the Fused Lasso and other total variation methods, multi-task learning methods and many more. In this paper, we present a general approach for computing the proximity operator of this class of regularizers, under the assumption that the proximity operator of the function ωis known in advance. Our approach builds on a recent line of research on optimal first order optimization methods and uses fixed point iterations for numerically computing the proximity operator. It is more general than current approaches and, as we show with numerical simulations, computationally more efficient than available first order methods which do not achieve the optimal rate. In particular, our method outperforms state of the art O(1/T) methods for overlapping Group Lasso and matches optimal O(1/T^2) methods for the Fused Lasso and tree structured Group Lasso.

cs.LG↗

Refinement of Operator-valued Reproducing Kernels

This paper studies the construction of a refinement kernel for a given operator-valued reproducing kernel such that the vector-valued reproducing kernel Hilbert space of the refinement kernel contains that of the given one as a subspace. The study is motivated from the need of updating the current operator-valued reproducing kernel in multi-task learning when underfitting or overfitting occurs. Numerical simulations confirm that the established refinement kernel method is able to meet this need. Various characterizations are provided based on feature maps and vector-valued integral representations of operator-valued reproducing kernels. Concrete examples of refining translation invariant and finite Hilbert-Schmidt operator-valued reproducing kernels are provided. Other examples include refinement of Hessian of scalar-valued translation-invariant kernels and transformation kernels. Existence and properties of operator-valued reproducing kernels preserved during the refinement process are also investigated.

cs.LG↗