arXiv subjects

Chang-Ock Lee

Publications and source records attributed to Chang-Ock Lee.

14 recordsLinked to original sources

arXiv2026

A Fully Dual Domain Decomposition Method for Extreme Learning Machines Using GenEO Spectral Coarse Spaces

Extreme learning machines (ELMs) are a class of machine learning algorithms that randomly intialize and subsequently fix the hidden layer parameters, and solve for the last layer coefficients using a least squares method. ELMs have been successfully applied to the solution of various partial differential equations, but the cost of solving large least squares problems limits their use in large scale problems or when high accuracy is desired. To address this limitation, domain decomposition methods (DDMs) have been developed for ELMs, reducing training times via parallel computation. Yet these methods, even those incorporating a coarse space, have struggled with scalability. This paper introduces a novel nonoverlapping DDM for ELMs that treats and enforces continuity and flux conditions equally. Additive Schwarz preconditioners and GenEO (Generalized Eigenvalues problems in the Overlaps) coarse spaces are constructed to accelerate the method. Numerical experiments show the new method outperforms previous DDMs for ELMs in both iteration count and accuracy, especially for the Stokes and wave equations. Furthermore, weak scalability has been observed in up to 4,096 subdomains when employing GenEO coarse spaces.

math.NA↗

arXiv2026

Hybrid Least Squares/Gradient Descent Methods for MIONets

In this paper, we propose an efficient hybrid least squares/gradient descent (LSGD) method for MIONets to accelerate training. This method generalizes the LSGD method for DeepONets. Since MIONet is the sum of the entrywise product of multiple branch networks and a trunk network, it can be viewed as a multilinear function with respect to the last layer parameters of each branch network. These sets of parameters can be optimized using the alternating least squares method, where we solve the LS system for a single branch network in turn. To handle the large-sized system matrix, we introduce Kronecker and Khatri-Rao products and tensor permutation matrices to factor the large matrix into small ones. Our method is compatible with a general type of $L^2$ loss with regularization terms for the last layer parameters of each branch, where linear operators can be applied to the MIONet output in each loss term.

cs.LG↗

arXiv2025

Hybrid Least Squares/Gradient Descent Methods for DeepONets

We propose an efficient hybrid least squares/gradient descent method to accelerate DeepONet training. Since the output of DeepONet can be viewed as linear with respect to the last layer parameters of the branch network, these parameters can be optimized using a least squares (LS) solve, and the remaining hidden layer parameters are updated by means of gradient descent form. However, building the LS system for all possible combinations of branch and trunk inputs yields a prohibitively large linear problem that is infeasible to solve directly. To address this issue, our method decomposes the large LS system into two smaller, more manageable subproblems $\unicode{x2014}$ one for the branch network and one for the trunk network $\unicode{x2014}$ and solves them separately. This method is generalized to a broader type of $L^2$ loss with a regularization term for the last layer parameters, including the case of unsupervised learning with physics-informed loss.

cs.LG↗

arXiv2025

A Neumann-Neumann Acceleration with Coarse Space for Domain Decomposition of Extreme Learning Machines

Extreme learning machines (ELMs), which preset hidden layer parameters and solve for last layer coefficients via a least squares method, can typically solve partial differential equations faster and more accurately than Physics Informed Neural Networks. However, they remain computationally expensive when high accuracy requires large least squares problems to be solved. Domain decomposition methods (DDMs) for ELMs have allowed parallel computation to reduce training times of large systems. This paper constructs a coarse space for ELMs, which enables further acceleration of their training. By partitioning interface variables into coarse and non-coarse variables, selective elimination introduces a Schur complement system on the non-coarse variables with the coarse problem embedded. Key to the performance of the proposed method is a Neumann-Neumann acceleration that utilizes the coarse space. Numerical experiments demonstrate significant speedup compared to a previous DDM method for ELMs.

math.NA↗

arXiv2024

A Nonoverlapping Domain Decomposition Method for Extreme Learning Machines: Elliptic Problems

Extreme learning machine (ELM) is a methodology for solving partial differential equations (PDEs) using a single hidden layer feed-forward neural network. It presets the weight/bias coefficients in the hidden layer with random values, which remain fixed throughout the computation, and uses a linear least squares method for training the parameters of the output layer of the neural network. It is known to be much faster than Physics informed neural networks. However, classical ELM is still computationally expensive when a high level of representation is desired in the solution as this requires solving a large least squares system. In this paper, we propose a nonoverlapping domain decomposition method (DDM) for ELMs that not only reduces the training time of ELMs, but is also suitable for parallel computation. In numerical analysis, DDMs have been widely studied to reduce the time to obtain finite element solutions for elliptic PDEs through parallel computation. Among these approaches, nonoverlapping DDMs are attracting the most attention. Motivated by these methods, we introduce local neural networks, which are valid only at corresponding subdomains, and an auxiliary variable at the interface. We construct a system on the variable and the parameters of local neural networks. A Schur complement system on the interface can be derived by eliminating the parameters of the output layer. The auxiliary variable is then directly obtained by solving the reduced system after which the parameters for each local neural network are solved in parallel. A method for initializing the hidden layer parameters suitable for high approximation quality in large systems is also proposed. Numerical results that verify the acceleration performance of the proposed method with respect to the number of subdomains are presented.

math.NA↗

arXiv2023

Balanced Group Convolution: An Improved Group Convolution Based on Approximability Estimates

The performance of neural networks has been significantly improved by increasing the number of channels in convolutional layers. However, this increase in performance comes with a higher computational cost, resulting in numerous studies focused on reducing it. One promising approach to address this issue is group convolution, which effectively reduces the computational cost by grouping channels. However, to the best of our knowledge, there has been no theoretical analysis on how well the group convolution approximates the standard convolution. In this paper, we mathematically analyze the approximation of the group convolution to the standard convolution with respect to the number of groups. Furthermore, we propose a novel variant of the group convolution called balanced group convolution, which shows a higher approximation with a small additional computational cost. We provide experimental results that validate our theoretical findings and demonstrate the superior performance of the balanced group convolution over other variants of group convolution.

cs.LG↗

arXiv2021

Two-level Group Convolution

Group convolution has been widely used in order to reduce the computation time of convolution, which takes most of the training time of convolutional neural networks. However, it is well known that a large number of groups significantly reduce the performance of group convolution. In this paper, we propose a new convolution methodology called ``two-level'' group convolution that is robust with respect to the increase of the number of groups and suitable for multi-GPU parallel computation. We first observe that the group convolution can be interpreted as a one-level block Jacobi approximation of the standard convolution, which is a popular notion in the field of numerical analysis. In numerical analysis, there have been numerous studies on the two-level method that introduces an intergroup structure that resolves the performance degradation issue without disturbing parallel computation. Motivated by these, we introduce a coarse-level structure which promotes intergroup communication without being a bottleneck in the group convolution. We show that all the additional work induced by the coarse-level structure can be efficiently processed in a distributed memory system. Numerical results that verify the robustness of the proposed method with respect to the number of groups are presented. Moreover, we compare the proposed method to various approaches for group convolution in order to highlight the superiority of the proposed method in terms of execution time, memory efficiency, and performance.

cs.LG↗

arXiv2021

Parareal Neural Networks Emulating a Parallel-in-time Algorithm

As deep neural networks (DNNs) become deeper, the training time increases. In this perspective, multi-GPU parallel computing has become a key tool in accelerating the training of DNNs. In this paper, we introduce a novel methodology to construct a parallel neural network that can utilize multiple GPUs simultaneously from a given DNN. We observe that layers of DNN can be interpreted as the time step of a time-dependent problem and can be parallelized by emulating a parallel-in-time algorithm called parareal. The parareal algorithm consists of fine structures which can be implemented in parallel and a coarse structure which gives suitable approximations to the fine structures. By emulating it, the layers of DNN are torn to form a parallel structure which is connected using a suitable coarse network. We report accelerated and accuracy-preserved results of the proposed methodology applied to VGG-16 and ResNet-1001 on several datasets.

math.NA↗

arXiv2020

Correction to: A dual iterative substructuring method with a small penalty parameter

In this corrigendum, we offer a correction to [J. Korean. Math. Soc., 54 (2017), pp. 461--477]. We construct a counterexample for the strengthened Cauchy--Schwarz inequality used in the original paper. In addition, we provide a new proof for Lemma 5 of the original paper, an estimate for the extremal eigenvalues of the standard unpreconditioned FETI-DP dual operator.

math.NA↗

arXiv2020

Optimal quadrature formulas for non-periodic functions in Sobolev space and its application to CT image reconstruction

In the present paper, optimal quadrature formulas in the sense of Sard are constructed for numerical integration of the integral $\int_a^be^{2πiωx}φ(x)d x$ with $ω\in \mathbb{R}$ in the Sobolev space $L_2^{(m)}[a,b]$ of complex-valued functions which are square integrable with $m$-th order derivative. Here, using the discrete analogue of the differential operator $\frac{d^{2m}}{d x^{2m}}$, the explicit formulas for optimal coefficients are obtained. The order of convergence of the obtained optimal quadrature formula is $O(h^m)$. As an application, we implement the filtered back-projection (FBP) algorithm, which is a well-known image reconstruction algorithm for computed tomography (CT). By approximating Fourier transforms and its inversion using the proposed optimal quadrature formula of the second and third orders, we observe that the accuracy of the reconstruction algorithm is improved. In numerical experiments, we compare the quality of the reconstructed image obtained by using the proposed optimal quadrature formulas with the conventional FBP, in which fast Fourier transform is used for the calculation of Fourier transform and its inversion. In the noise test, the proposed algorithm provides more reliable results against the noise than the conventional FBP.

math.NA↗

arXiv2019

A Finite Element Nonoverlapping Domain Decomposition Method with Lagrange Multipliers for the Dual Total Variation Minimizations

In this paper, we consider a primal-dual domain decomposition method for total variation regularized problems appearing in mathematical image processing. The model problem is transformed into an equivalent constrained minimization problem by tearing-and-interconnecting domain decomposition. Then, the continuity constraints on the subdomain interfaces are treated by introducing Lagrange multipliers. The resulting saddle point problem is solved by the first order primal-dual algorithm. We apply the proposed method to image denoising, inpainting, and segmentation problems with either $L^2$-fidelity or $L^1$-fidelity. Numerical results show that the proposed method outperforms the existing state-of-the-art methods.

math.NA↗

arXiv2019

Fast Nonoverlapping Block Jacobi Method for the Dual Rudin--Osher--Fatemi Model

We consider nonoverlapping domain decomposition methods for the Rudin--Osher--Fatemi~(ROF) model, which is one of the standard models in mathematical image processing. The image domain is partitioned into rectangular subdomains and local problems in subdomains are solved in parallel. Local problems can adopt existing state-of-the-art solvers for the ROF model. We show that the nonoverlapping relaxed block Jacobi method for a dual formulation of the ROF model has the $O(1/n)$ convergence rate of the energy functional, where $n$ is the number of iterations. Moreover, by exploiting the forward-backward splitting structure of the method, we propose an accelerated version whose convergence rate is $O(1/n^2)$. The proposed method converges faster than existing domain decomposition methods both theoretically and practically, while the main computational cost of each iteration remains the same. We also provide the dependence of the convergence rates of the block Jacobi methods on the image size and the number of subdomains. Numerical results for comparisons with existing methods are presented.

math.NA↗

arXiv2019

On an optimal quadrature formula for approximation of Fourier integrals in the space $L_2^{(1)}$

This paper deals with the construction of an optimal quadrature formula for the approximation of Fourier integrals in the Sobolev space $L_2^{(1)}[a,b]$ of non-periodic, complex valued functions which are square integrable with first order derivative. Here the quadrature sum consists of linear combination of the given function values in a uniform grid. The difference between the integral and the quadrature sum is estimated by the norm of the error functional. The optimal quadrature formula is obtained by minimizing the norm of the error functional with respect to coefficients. Analytic formulas for optimal coefficients can also be obtained using discrete analogue of the differential operator $d^2/d x^2$. In addition, the convergence order of the optimal quadrature formula is studied. It is proved that the obtained formula is exact for all linear polynomials. Thus, it is shown that the convergence order of the optimal quadrature formula for functions of the space $C^2[a,b]$ is $O(h^2)$. Moreover, several numerical results are presented and the obtained optimal quadrature formula is applied to reconstruct the X-ray Computed Tomography image by approximating Fourier transforms.

math.NA↗

arXiv2019

A Finite Element Approach for the Dual Rudin--Osher--Fatemi Model and Its Nonoverlapping Domain Decomposition Methods

We consider a finite element discretization for the dual Rudin--Osher--Fatemi model using a Raviart--Thomas basis for $H_0 (\mathrm{div};Ω)$. Since the proposed discretization has splitting property for the energy functional, which is not satisfied for existing finite difference based discretizations, it is more adequate for designing domain decomposition methods. In this paper, a primal domain decomposition method is proposed, which resembles the classical Schur complement method for the second order elliptic problems, and it achieves $O(1/n^2)$ convergence. A primal-dual domain decomposition method based on the method of Lagrange multipliers on the subdomain interfaces is also considered. Local problems of the proposed primal-dual domain decomposition method can be solved in linear convergence rate. Numerical results for the proposed methods are provided.

math.NA↗