arXiv subjects

Youngjoon Hong

Publications and source records attributed to Youngjoon Hong.

At least 37 records · Page 2Linked to original sources

arXiv2024

MaD-Scientist: AI-based Scientist solving Convection-Diffusion-Reaction Equations Using Massive PINN-Based Prior Data

Large language models (LLMs), like ChatGPT, have shown that even trained with noisy prior data, they can generalize effectively to new tasks through in-context learning (ICL) and pre-training techniques. Motivated by this, we explore whether a similar approach can be applied to scientific foundation models (SFMs). Our methodology is structured as follows: (i) we collect low-cost physics-informed neural network (PINN)-based approximated prior data in the form of solutions to partial differential equations (PDEs) constructed through an arbitrary linear combination of mathematical dictionaries; (ii) we utilize Transformer architectures with self and cross-attention mechanisms to predict PDE solutions without knowledge of the governing equations in a zero-shot setting; (iii) we provide experimental evidence on the one-dimensional convection-diffusion-reaction equation, which demonstrate that pre-training remains robust even with approximated prior data, with only marginal impacts on test accuracy. Notably, this finding opens the path to pre-training SFMs with realistic, low-cost data instead of (or in conjunction with) numerical high-cost data. These results support the conjecture that SFMs can improve in a manner similar to LLMs, where fully cleaning the vast set of sentences crawled from the Internet is nearly impossible.

cs.LG↗

arXiv2024

Singular Layer Physics-Informed Neural Network Method for Convection-Dominated Boundary Layer Problems in Two Dimensions

This research explores neural network-based numerical approximation of two-dimensional convection-dominated singularly perturbed problems on square, circular, and elliptic domains. Singularly perturbed boundary value problems pose significant challenges due to sharp boundary layers in their solutions. Additionally, the characteristic points of these domains give rise to degenerate boundary layer problems. The stiffness of these problems, caused by sharp singular layers, can lead to substantial computational errors if not properly addressed. Conventional neural network-based approaches often fail to capture these sharp transitions accurately, highlighting a critical flaw in machine learning methods. To address these issues, we conduct a thorough boundary layer analysis to enhance our understanding of sharp transitions within the boundary layers, guiding the application of numerical methods. Specifically, we employ physics-informed neural networks (PINNs) to better handle these boundary layer problems. However, PINNs may struggle with rapidly varying singularly perturbed solutions in small domain regions, leading to inaccurate or unstable results. To overcome this limitation, we introduce a semi-analytic method that augments PINNs with singular layers or corrector functions. Our numerical experiments demonstrate significant improvements in both accuracy and stability, showcasing the effectiveness of our proposed approach.

math.NA↗

arXiv2024

Simple arithmetic operation in latent space can generate a novel three dimensional graph metamaterials

Recent advancements in artificial intelligence (AI)-based design strategies for metamaterials have revolutionized the creation of customizable architectures spanning nano- to macro-scale dimensions, achieving unprecedented mechanical behaviors that surpass the inherent properties of the constituent materials. However, the growing complexity of these methods poses challenges in generating diverse metamaterials without substantial human and computational resources, hindering widespread adoption. Addressing this, our study introduces an innovative design strategy capable of generating various three-dimensional graph metamaterials using simple arithmetic operations within the latent space. By seamlessly integrating hidden representations of disentangled latent space and latent diffusion processes, our approach provides a comprehensive understanding of complex design spaces, generating diverse graph metamaterials through arithmetic operations. This methodology stands as a versatile tool for creating structures ranging from repetitive lattice structures to functionally graded mechanical metamaterials. It also serves as an inverse design strategy for diverse lattice structures, including crystalline structures and those made of trabecular bone. We believe that this methodology represents a foundational step in advancing our comprehension of the intricate latent design space, offering the potential to establish a unified model for various traditional generative models in the realm of mechanical metamaterials.

physics.app-ph↗

arXiv2024

Error analysis for finite element operator learning methods for solving parametric second-order elliptic PDEs

In this paper, we provide a theoretical analysis of a type of operator learning method without data reliance based on the classical finite element approximation, which is called the finite element operator network (FEONet). We first establish the convergence of this method for general second-order linear elliptic PDEs with respect to the parameters for neural network approximation. In this regard, we address the role of the condition number of the finite element matrix in the convergence of the method. Secondly, we derive an explicit error estimate for the self-adjoint case. For this, we investigate some regularity properties of the solution in certain function classes for a neural network approximation, verifying the sufficient condition for the solution to have the desired regularity. Finally, we will also conduct some numerical experiments that support the theoretical findings, confirming the role of the condition number of the finite element matrix in the overall convergence.

math.NA↗

arXiv2024

Separable Physics-informed Neural Networks for Solving the BGK Model of the Boltzmann Equation

In this study, we introduce a method based on Separable Physics-Informed Neural Networks (SPINNs) for effectively solving the BGK model of the Boltzmann equation. While the mesh-free nature of PINNs offers significant advantages in handling high-dimensional partial differential equations (PDEs), challenges arise when applying quadrature rules for accurate integral evaluation in the BGK operator, which can compromise the mesh-free benefit and increase computational costs. To address this, we leverage the canonical polyadic decomposition structure of SPINNs and the linear nature of moment calculation, achieving a substantial reduction in computational expense for quadrature rule application. The multi-scale nature of the particle density function poses difficulties in precisely approximating macroscopic moments using neural networks. To improve SPINN training, we introduce the integration of Gaussian functions into SPINNs, coupled with a relative loss approach. This modification enables SPINNs to decay as rapidly as Maxwellian distributions, thereby enhancing the accuracy of macroscopic moment approximations. The relative loss design further ensures that both large and small-scale features are effectively captured by the SPINNs. The efficacy of our approach is demonstrated through a series of five numerical experiments, including the solution to a challenging 3D Riemann problem. These results highlight the potential of our novel method in efficiently and accurately addressing complex challenges in computational physics.

math.NA↗

arXiv2023

Semi-analytic PINN methods for boundary layer problems in a rectangular domain

Singularly perturbed boundary value problems pose a significant challenge for their numerical approximations because of the presence of sharp boundary layers. These sharp boundary layers are responsible for the stiffness of solutions, which leads to large computational errors, if not properly handled. It is well-known that the classical numerical methods as well as the Physics-Informed Neural Networks (PINNs) require some special treatments near the boundary, e.g., using extensive mesh refinements or finer collocation points, in order to obtain an accurate approximate solution especially inside of the stiff boundary layer. In this article, we modify the PINNs and construct our new semi-analytic SL-PINNs suitable for singularly perturbed boundary value problems. Performing the boundary layer analysis, we first find the corrector functions describing the singular behavior of the stiff solutions inside boundary layers. Then we obtain the SL-PINN approximations of the singularly perturbed problems by embedding the explicit correctors in the structure of PINNs or by training the correctors together with the PINN approximations. Our numerical experiments confirm that our new SL-PINN methods produce stable and accurate approximations for stiff solutions.

math.NA↗

arXiv2023

Physics-Informed Convolutional Transformer for Predicting Volatility Surface

Predicting volatility is important for asset predicting, option pricing and hedging strategies because it cannot be directly observed in the financial market. The Black-Scholes option pricing model is one of the most widely used models by market participants. Notwithstanding, the Black-Scholes model is based on heavily criticized theoretical premises, one of which is the constant volatility assumption. The dynamics of the volatility surface is difficult to estimate. In this paper, we establish a novel architecture based on physics-informed neural networks and convolutional transformers. The performance of the new architecture is directly compared to other well-known deep-learning architectures, such as standard physics-informed neural networks, convolutional long-short term memory (ConvLSTM), and self-attention ConvLSTM. Numerical evidence indicates that the proposed physics-informed convolutional transformer network achieves a superior performance than other methods.

q-fin.CP↗

arXiv2023

Singular layer Physics Informed Neural Network method for Plane Parallel Flows

We construct in this article the semi-analytic Physics Informed Neural Networks (PINNs), called {\em singular layer PINNs} (or {\em sl-PINNs}), that are suitable to predict the stiff solutions of plane-parallel flows at a small viscosity. Recalling the boundary layer analysis, we first find the corrector for the problem which describes the singular behavior of the viscous flow inside boundary layers. Then, using the components of the corrector and its curl, we build our new {\em sl-PINN} predictions for the velocity and the vorticity by either embedding the explicit expression of the corrector (or its curl) in the structure of PINNs or by training the implicit parts of the corrector (or its curl) together with the PINN predictions. Numerical experiments confirm that our new {\em sl-PINNs} produce stable and accurate predicted solutions for the plane-parallel flows at a small viscosity.

math.NA↗

arXiv2023

Separable PINN: Mitigating the Curse of Dimensionality in Physics-Informed Neural Networks

Physics-informed neural networks (PINNs) have emerged as new data-driven PDE solvers for both forward and inverse problems. While promising, the expensive computational costs to obtain solutions often restrict their broader applicability. We demonstrate that the computations in automatic differentiation (AD) can be significantly reduced by leveraging forward-mode AD when training PINN. However, a naive application of forward-mode AD to conventional PINNs results in higher computation, losing its practical benefit. Therefore, we propose a network architecture, called separable PINN (SPINN), which can facilitate forward-mode AD for more efficient computation. SPINN operates on a per-axis basis instead of point-wise processing in conventional PINNs, decreasing the number of network forward passes. Besides, while the computation and memory costs of standard PINNs grow exponentially along with the grid resolution, that of our model is remarkably less susceptible, mitigating the curse of dimensionality. We demonstrate the effectiveness of our model in various PDE systems by significantly reducing the training run-time while achieving comparable accuracy. Project page: https://jwcho5576.github.io/spinn/

cs.LG↗

arXiv2023

Separable Physics-Informed Neural Networks

Physics-informed neural networks (PINNs) have recently emerged as promising data-driven PDE solvers showing encouraging results on various PDEs. However, there is a fundamental limitation of training PINNs to solve multi-dimensional PDEs and approximate highly complex solution functions. The number of training points (collocation points) required on these challenging PDEs grows substantially, but it is severely limited due to the expensive computational costs and heavy memory overhead. To overcome this issue, we propose a network architecture and training algorithm for PINNs. The proposed method, separable PINN (SPINN), operates on a per-axis basis to significantly reduce the number of network propagations in multi-dimensional PDEs unlike point-wise processing in conventional PINNs. We also propose using forward-mode automatic differentiation to reduce the computational cost of computing PDE residuals, enabling a large number of collocation points (>10^7) on a single commodity GPU. The experimental results show drastically reduced computational costs (62x in wall-clock time, 1,394x in FLOPs given the same number of collocation points) in multi-dimensional PDEs while achieving better accuracy. Furthermore, we present that SPINN can solve a chaotic (2+1)-d Navier-Stokes equation significantly faster than the best-performing prior method (9 minutes vs 10 hours in a single GPU), maintaining accuracy. Finally, we showcase that SPINN can accurately obtain the solution of a highly nonlinear and multi-dimensional PDE, a (3+1)-d Navier-Stokes equation. For visualized results and code, please see https://jwcho5576.github.io/spinn.github.io/.

cs.LG↗

arXiv2023

Spectral operator learning for parametric PDEs without data reliance

In this paper, we introduce the Spectral Coefficient Learning via Operator Network (SCLON), a novel operator learning-based approach for solving parametric partial differential equations (PDEs) without the need for data harnessing. The cornerstone of our method is the spectral methodology that employs expansions using orthogonal functions, such as Fourier series and Legendre polynomials, enabling accurate PDE solutions with fewer grid points. By merging the merits of spectral methods - encompassing high accuracy, efficiency, generalization, and the exact fulfillment of boundary conditions - with the prowess of deep neural networks, SCLON offers a transformative strategy. Our approach not only eliminates the need for paired input-output training data, which typically requires extensive numerical computations, but also effectively learns and predicts solutions of complex parametric PDEs, ranging from singularly perturbed convection-diffusion equations to the Navier-Stokes equations. The proposed framework demonstrates superior performance compared to existing scientific machine learning techniques, offering solutions for multiple instances of parametric PDEs without harnessing data. The mathematical framework is robust and reliable, with a well-developed loss function derived from the weak formulation, ensuring accurate approximation of solutions while exactly satisfying boundary conditions. The method's efficacy is further illustrated through its ability to accurately predict intricate natural behaviors like the Kolmogorov flow and boundary layers. In essence, our work pioneers a compelling avenue for parametric PDE solutions, serving as a bridge between traditional numerical methodologies and cutting-edge machine learning techniques in the realm of scientific computation.

cs.LG↗

arXiv2023

Hydra: Multi-head Low-rank Adaptation for Parameter Efficient Fine-tuning

The recent surge in large-scale foundation models has spurred the development of efficient methods for adapting these models to various downstream tasks. Low-rank adaptation methods, such as LoRA, have gained significant attention due to their outstanding parameter efficiency and no additional inference latency. This paper investigates a more general form of adapter module based on the analysis that parallel and sequential adaptation branches learn novel and general features during fine-tuning, respectively. The proposed method, named Hydra, due to its multi-head computational branches, combines parallel and sequential branch to integrate capabilities, which is more expressive than existing single branch methods and enables the exploration of a broader range of optimal points in the fine-tuning process. In addition, the proposed adaptation method explicitly leverages the pre-trained weights by performing a linear combination of the pre-trained features. It allows the learned features to have better generalization performance across diverse downstream tasks. Furthermore, we perform a comprehensive analysis of the characteristics of each adaptation branch with empirical evidence. Through an extensive range of experiments, encompassing comparisons and ablation studies, we substantiate the efficiency and demonstrate the superior performance of Hydra. This comprehensive evaluation underscores the potential impact and effectiveness of Hydra in a variety of applications. Our code is available on \url{https://github.com/extremebird/Hydra}

cs.CV↗

arXiv2023

An Adaptive Dual-level Reinforcement Learning Approach for Optimal Trade Execution

The purpose of this research is to devise a tactic that can closely track the daily cumulative volume-weighted average price (VWAP) using reinforcement learning. Previous studies often choose a relatively short trading horizon to implement their models, making it difficult to accurately track the daily cumulative VWAP since the variations of financial data are often insignificant within the short trading horizon. In this paper, we aim to develop a strategy that can accurately track the daily cumulative VWAP while minimizing the deviation from the VWAP. We propose a method that leverages the U-shaped pattern of intraday stock trade volumes and use Proximal Policy Optimization (PPO) as the learning algorithm. Our method follows a dual-level approach: a Transformer model that captures the overall(global) distribution of daily volumes in a U-shape, and a LSTM model that handles the distribution of orders within smaller(local) time intervals. The results from our experiments suggest that this dual-level architecture improves the accuracy of approximating the cumulative VWAP, when compared to previous reinforcement learning-based models.

q-fin.CP↗

arXiv2023

PIXEL: Physics-Informed Cell Representations for Fast and Accurate PDE Solvers

With the increases in computational power and advances in machine learning, data-driven learning-based methods have gained significant attention in solving PDEs. Physics-informed neural networks (PINNs) have recently emerged and succeeded in various forward and inverse PDE problems thanks to their excellent properties, such as flexibility, mesh-free solutions, and unsupervised training. However, their slower convergence speed and relatively inaccurate solutions often limit their broader applicability in many science and engineering domains. This paper proposes a new kind of data-driven PDEs solver, physics-informed cell representations (PIXEL), elegantly combining classical numerical methods and learning-based approaches. We adopt a grid structure from the numerical methods to improve accuracy and convergence speed and overcome the spectral bias presented in PINNs. Moreover, the proposed method enjoys the same benefits in PINNs, e.g., using the same optimization frameworks to solve both forward and inverse PDE problems and readily enforcing PDE constraints with modern automatic differentiation techniques. We provide experimental results on various challenging PDEs that the original PINNs have struggled with and show that PIXEL achieves fast convergence speed and high accuracy. Project page: https://namgyukang.github.io/PIXEL/

cs.LG↗

arXiv2022

Unsupervised Legendre-Galerkin Neural Network for Singularly Perturbed Partial Differential Equations

Machine learning methods have been lately used to solve partial differential equations (PDEs) and dynamical systems. These approaches have been developed into a novel research field known as scientific machine learning in which techniques such as deep neural networks and statistical learning are applied to classical problems of applied mathematics. In this paper, we develop a novel numerical algorithm that incorporates machine learning and artificial intelligence to solve PDEs. Based on the Legendre-Galerkin framework, we propose the {\it unsupervised machine learning} algorithm to learn {\it multiple instances} of the solutions for different types of PDEs. Our approach overcomes the limitations of data-driven and physics-based methods. The proposed neural network is applied to general 1D and 2D PDEs with various boundary conditions as well as convection-dominated {\it singularly perturbed PDEs} that exhibit strong boundary layer behavior.

cs.LG↗

arXiv2022

Convergence analysis of unsupervised Legendre-Galerkin neural networks for linear second-order elliptic PDEs

In this paper, we perform the convergence analysis of unsupervised Legendre--Galerkin neural networks (ULGNet), a deep-learning-based numerical method for solving partial differential equations (PDEs). Unlike existing deep learning-based numerical methods for PDEs, the ULGNet expresses the solution as a spectral expansion with respect to the Legendre basis and predicts the coefficients with deep neural networks by solving a variational residual minimization problem. Since the corresponding loss function is equivalent to the residual induced by the linear algebraic system depending on the choice of basis functions, we prove that the minimizer of the discrete loss function converges to the weak solution of the PDEs. Numerical evidence will also be provided to support the theoretical result. Key technical tools include the variant of the universal approximation theorem for bounded neural networks, the analysis of the stiffness and mass matrices, and the uniform law of large numbers in terms of the Rademacher complexity.

math.NA↗

arXiv2022

Invertible Monotone Operators for Normalizing Flows

Normalizing flows model probability distributions by learning invertible transformations that transfer a simple distribution into complex distributions. Since the architecture of ResNet-based normalizing flows is more flexible than that of coupling-based models, ResNet-based normalizing flows have been widely studied in recent years. Despite their architectural flexibility, it is well-known that the current ResNet-based models suffer from constrained Lipschitz constants. In this paper, we propose the monotone formulation to overcome the issue of the Lipschitz constants using monotone operators and provide an in-depth theoretical analysis. Furthermore, we construct an activation function called Concatenated Pila (CPila) to improve gradient flow. The resulting model, Monotone Flows, exhibits an excellent performance on multiple density estimation benchmarks (MNIST, CIFAR-10, ImageNet32, ImageNet64). Code is available at https://github.com/mlvlab/MonotoneFlows.

cs.LG↗

arXiv2022

Semi-analytic PINN methods for singularly perturbed boundary value problems

We propose a new semi-analytic physics informed neural network (PINN) to solve singularly perturbed boundary value problems. The PINN is a scientific machine learning framework that offers a promising perspective for finding numerical solutions to partial differential equations. The PINNs have shown impressive performance in solving various differential equations including time-dependent and multi-dimensional equations involved in a complex geometry of the domain. However, when considering stiff differential equations, neural networks in general fail to capture the sharp transition of solutions, due to the spectral bias. To resolve this issue, here we develop the semi-analytic PINN methods, enriched by using the so-called corrector functions obtained from the boundary layer analysis. Our new enriched PINNs accurately predict numerical solutions to the singular perturbation problems. Numerical experiments include various types of singularly perturbed linear and nonlinear differential equations.

math.NA↗