arXiv subjects

Alireza Doostan

Publications and source records attributed to Alireza Doostan.

At least 37 records · Page 2Linked to original sources

arXiv2024

Constrained or Unconstrained? Neural-Network-Based Equation Discovery from Data

Throughout many fields, practitioners often rely on differential equations to model systems. Yet, for many applications, the theoretical derivation of such equations and/or accurate resolution of their solutions may be intractable. Instead, recently developed methods, including those based on parameter estimation, operator subset selection, and neural networks, allow for the data-driven discovery of both ordinary and partial differential equations (PDEs), on a spectrum of interpretability. The success of these strategies is often contingent upon the correct identification of representative equations from noisy observations of state variables and, as importantly and intertwined with that, the mathematical strategies utilized to enforce those equations. Specifically, the latter has been commonly addressed via unconstrained optimization strategies. Representing the PDE as a neural network, we propose to discover the PDE by solving a constrained optimization problem and using an intermediate state representation similar to a Physics-Informed Neural Network (PINN). The objective function of this constrained optimization problem promotes matching the data, while the constraints require that the PDE is satisfied at several spatial collocation points. We present a penalty method and a widely used trust-region barrier method to solve this constrained optimization problem, and we compare these methods on numerical examples. Our results on the Burgers' and the Korteweg-De Vreis equations demonstrate that the latter constrained method outperforms the penalty method, particularly for higher noise levels or fewer collocation points. For both methods, we solve these discovered neural network PDEs with classical methods, such as finite difference methods, as opposed to PINNs-type methods relying on automatic differentiation. We briefly highlight other small, yet crucial, implementation details.

cs.LG↗

arXiv2024

Online randomized interpolative decomposition with a posteriori error estimator for temporal PDE data reduction

Traditional low-rank approximation is a powerful tool to compress the huge data matrices that arise in simulations of partial differential equations (PDE), but suffers from high computational cost and requires several passes over the PDE data. The compressed data may also lack interpretability thus making it difficult to identify feature patterns from the original data. To address these issues, we present an online randomized algorithm to compute the interpolative decomposition (ID) of large-scale data matrices {\em in situ}. Compared to previous randomized IDs that used the QR decomposition to determine the column basis, we adopt a streaming ridge leverage score-based column subset selection algorithm that dynamically selects proper basis columns from the data and thus avoids an extra pass over the data to compute the coefficient matrix of the ID. In particular, we adopt a single-pass error estimator based on the non-adaptive Hutch++ algorithm to provide real-time error approximation for determining the best coefficients. As a result, our approach only needs a single pass over the original data and thus is suitable for large and high-dimensional matrices stored outside of core memory or generated in PDE simulations. A strategy to improve the accuracy of the reconstructed data gradient, when desired, within the ID framework is also presented. We provide numerical experiments on turbulent channel flow and ignition simulations, and on the NSTX Gas Puff Image dataset, comparing our algorithm with the offline ID algorithm to demonstrate its utility in real-world applications.

math.NA↗

arXiv2023

Bi-fidelity Variational Auto-encoder for Uncertainty Quantification

Quantifying the uncertainty of quantities of interest (QoIs) from physical systems is a primary objective in model validation. However, achieving this goal entails balancing the need for computational efficiency with the requirement for numerical accuracy. To address this trade-off, we propose a novel bi-fidelity formulation of variational auto-encoders (BF-VAE) designed to estimate the uncertainty associated with a QoI from low-fidelity (LF) and high-fidelity (HF) samples of the QoI. This model allows for the approximation of the statistics of the HF QoI by leveraging information derived from its LF counterpart. Specifically, we design a bi-fidelity auto-regressive model in the latent space that is integrated within the VAE's probabilistic encoder-decoder structure. An effective algorithm is proposed to maximize the variational lower bound of the HF log-likelihood in the presence of limited HF data, resulting in the synthesis of HF realizations with a reduced computational cost. Additionally, we introduce the concept of the bi-fidelity information bottleneck (BF-IB) to provide an information-theoretic interpretation of the proposed BF-VAE model. Our numerical results demonstrate that BF-VAE leads to considerably improved accuracy, as compared to a VAE trained using only HF data, when limited HF data is available.

stat.ML↗

arXiv2023

QuadConv: Quadrature-Based Convolutions with Applications to Non-Uniform PDE Data Compression

We present a new convolution layer for deep learning architectures which we call QuadConv -- an approximation to continuous convolution via quadrature. Our operator is developed explicitly for use on non-uniform, mesh-based data, and accomplishes this by learning a continuous kernel that can be sampled at arbitrary locations. Moreover, the construction of our operator admits an efficient implementation which we detail and construct. As an experimental validation of our operator, we consider the task of compressing partial differential equation (PDE) simulation data from fixed meshes. We show that QuadConv can match the performance of standard discrete convolutions on uniform grid data by comparing a QuadConv autoencoder (QCAE) to a standard convolutional autoencoder (CAE). Further, we show that the QCAE can maintain this accuracy even on non-uniform data. In both cases, QuadConv also outperforms alternative unstructured convolution methods such as graph convolution.

cs.LG↗

arXiv2023

In Situ Framework for Coupling Simulation and Machine Learning with Application to CFD

Recent years have seen many successful applications of machine learning (ML) to facilitate fluid dynamic computations. As simulations grow, generating new training datasets for traditional offline learning creates I/O and storage bottlenecks. Additionally, performing inference at runtime requires non-trivial coupling of ML framework libraries with simulation codes. This work offers a solution to both limitations by simplifying this coupling and enabling in situ training and inference workflows on heterogeneous clusters. Leveraging SmartSim, the presented framework deploys a database to store data and ML models in memory, thus circumventing the file system. On the Polaris supercomputer, we demonstrate perfect scaling efficiency to the full machine size of the data transfer and inference costs thanks to a novel co-located deployment of the database. Moreover, we train an autoencoder in situ from a turbulent flow simulation, showing that the framework overhead is negligible relative to a solver time step and training epoch.

cs.LG↗

arXiv2023

Derivative-based SINDy (DSINDy): Addressing the challenge of discovering governing equations from noisy data

Recent advances in the field of data-driven dynamics allow for the discovery of ODE systems using state measurements. One approach, known as Sparse Identification of Nonlinear Dynamics (SINDy), assumes the dynamics are sparse within a predetermined basis in the states and finds the expansion coefficients through linear regression with sparsity constraints. This approach requires an accurate estimation of the state time derivatives, which is not necessarily possible in the high-noise regime without additional constraints. We present an approach called Derivative-based SINDy (DSINDy) that combines two novel methods to improve ODE recovery at high noise levels. First, we denoise the state variables by applying a projection operator that leverages the assumed basis for the system dynamics. Second, we use a second order cone program (SOCP) to find the derivative and governing equations simultaneously. We derive theoretical results for the projection-based denoising step, which allow us to estimate the values of hyperparameters used in the SOCP formulation. This underlying theory helps limit the number of required user-specified parameters. We present results demonstrating that our approach leads to improved system recovery for the Van der Pol oscillator, the Duffing oscillator, and the Rössler attractor.

math.DS↗

arXiv2023

Simultaneous Identification and Denoising of Dynamical Systems

In recent years there has been a push to discover the governing equations dynamical systems directly from measurements of the state, often motivated by systems that are too complex to directly model. Although there has been substantial work put into such a discovery, doing so in the case of large noise has proved challenging. Here we develop an algorithm for Simultaneous Identification and Denoising of a Dynamical System (SIDDS). We infer the noise in the state measurements by requiring that the denoised data satisfies the dynamical system with an equality constraint. This is unlike existing work where the mismatch in the dynamical system is treated as a penalty in the objective. We assume the dynamics is represented in a pre-defined basis and develop a sequential quadratic programming approach to solve the SIDDS problem featuring a direct solution of KKT system with a specialized preconditioner. In addition, we show how we can include sparsity promoting regularization using an iteratively reweighted least squares approach. The resulting algorithm leads to estimates of the dynamical system that approximately achieve the Cramér-Rao lower bound and, with sparsity promotion, can correctly identify the sparsity structure for higher levels of noise than existing techniques. Moreover, because SIDDS decouples the data from the evolution of the dynamical system, we show how to modify the problem to accurately identify systems from low sample rate measurements. The inverse problem approach and solution framework used by SIDDS has the potential to be expanded to related problems identifying governing equations from noisy data.

math.OC↗

arXiv2022

Quadrature Sampling of Parametric Models with Bi-fidelity Boosting

Least squares regression is a ubiquitous tool for building emulators (a.k.a. surrogate models) of problems across science and engineering for purposes such as design space exploration and uncertainty quantification. When the regression data are generated using an experimental design process (e.g., a quadrature grid) involving computationally expensive models, or when the data size is large, sketching techniques have shown promise to reduce the cost of the construction of the regression model while ensuring accuracy comparable to that of the full data. However, random sketching strategies, such as those based on leverage scores, lead to regression errors that are random and may exhibit large variability. To mitigate this issue, we present a novel boosting approach that leverages cheaper, lower-fidelity data of the problem at hand to identify the best sketch among a set of candidate sketches. This in turn specifies the sketch of the intended high-fidelity model and the associated data. We provide theoretical analyses of this bi-fidelity boosting (BFB) approach and discuss the conditions the low- and high-fidelity data must satisfy for a successful boosting. In doing so, we derive a bound on the residual norm of the BFB sketched solution relating it to its ideal, but computationally expensive, high-fidelity boosted counterpart. Empirical results on both manufactured and PDE data corroborate the theoretical analyses and illustrate the efficacy of the BFB solution in reducing the regression error, as compared to the non-boosted solution.

math.NA↗

arXiv2022

Bi-fidelity Modeling of Uncertain and Partially Unknown Systems using DeepONets

Recent advances in modeling large-scale complex physical systems have shifted research focuses towards data-driven techniques. However, generating datasets by simulating complex systems can require significant computational resources. Similarly, acquiring experimental datasets can prove difficult as well. For these systems, often computationally inexpensive, but in general inaccurate, models, known as the low-fidelity models, are available. In this paper, we propose a bi-fidelity modeling approach for complex physical systems, where we model the discrepancy between the true system's response and low-fidelity response in the presence of a small training dataset from the true system's response using a deep operator network (DeepONet), a neural network architecture suitable for approximating nonlinear operators. We apply the approach to model systems that have parametric uncertainty and are partially unknown. Three numerical examples are used to show the efficacy of the proposed approach to model uncertain and partially unknown complex physical systems.

stat.ML↗

arXiv2022

A Priori Denoising Strategies for Sparse Identification of Nonlinear Dynamical Systems: A Comparative Study

In recent years, identification of nonlinear dynamical systems from data has become increasingly popular. Sparse regression approaches, such as Sparse Identification of Nonlinear Dynamics (SINDy), fostered the development of novel governing equation identification algorithms assuming the state variables are known a priori and the governing equations lend themselves to sparse, linear expansions in a (nonlinear) basis of the state variables. In the context of the identification of governing equations of nonlinear dynamical systems, one faces the problem of identifiability of model parameters when state measurements are corrupted by noise. Measurement noise affects the stability of the recovery process yielding incorrect sparsity patterns and inaccurate estimation of coefficients of the governing equations. In this work, we investigate and compare the performance of several local and global smoothing techniques to a priori denoise the state measurements and numerically estimate the state time-derivatives to improve the accuracy and robustness of two sparse regression methods to recover governing equations: Sequentially Thresholded Least Squares (STLS) and Weighted Basis Pursuit Denoising (WBPDN) algorithms. We empirically show that, in general, global methods, which use the entire measurement data set, outperform local methods, which employ a neighboring data subset around a local point. We additionally compare Generalized Cross Validation (GCV) and Pareto curve criteria as model selection techniques to automatically estimate near optimal tuning parameters, and conclude that Pareto curves yield better results. The performance of the denoising strategies and sparse regression methods is empirically evaluated through well-known benchmark problems of nonlinear dynamical systems.

stat.ML↗

arXiv2022

Automated processing of X-ray computed tomography images via panoptic segmentation for modeling woven composite textiles

A new, machine learning-based approach for automatically generating 3D digital geometries of woven composite textiles is proposed to overcome the limitations of existing analytical descriptions and segmentation methods. In this approach, panoptic segmentation is leveraged to produce instance segmented semantic masks from X-ray computed tomography (CT) images. This effort represents the first deep learning based automated process for segmenting unique yarn instances in a woven composite textile. Furthermore, it improves on existing methods by providing instance-level segmentation on low contrast CT datasets. Frame-to-frame instance tracking is accomplished via an intersection-over-union (IoU) approach adopted from video panoptic segmentation for assembling a 3D geometric model. A corrective recognition algorithm is developed to improve the recognition quality (RQ). The panoptic quality (PQ) metric is adopted to provide a new universal evaluation metric for reconstructed woven composite textiles. It is found that the panoptic segmentation network generalizes well to new CT images that are similar to the training set but does not extrapolate well to CT images of differing geometry, texture, and contrast. The utility of this approach is demonstrated by capturing yarn flow directions, contact regions between individual yarns, and the spatially varying cross-sectional areas of the yarns.

cs.CV↗

arXiv2022

GenMod: A generative modeling approach for spectral representation of PDEs with random inputs

We propose a method for quantifying uncertainty in high-dimensional PDE systems with random parameters, where the number of solution evaluations is small. Parametric PDE solutions are often approximated using a spectral decomposition based on polynomial chaos expansions. For the class of systems we consider (i.e., high dimensional with limited solution evaluations) the coefficients are given by an underdetermined linear system in a regression formulation. This implies additional assumptions, such as sparsity of the coefficient vector, are needed to approximate the solution. Here, we present an approach where we assume the coefficients are close to the range of a generative model that maps from a low to a high dimensional space of coefficients. Our approach is inspired be recent work examining how generative models can be used for compressed sensing in systems with random Gaussian measurement matrices. Using results from PDE theory on coefficient decay rates, we construct an explicit generative model that predicts the polynomial chaos coefficient magnitudes. The algorithm we developed to find the coefficients, which we call GenMod, is composed of two main steps. First, we predict the coefficient signs using Orthogonal Matching Pursuit. Then, we assume the coefficients are within a sparse deviation from the range of a sign-adjusted generative model. This allows us to find the coefficients by solving a nonconvex optimization problem, over the input space of the generative model and the space of sparse vectors. We obtain theoretical recovery results for a Lipschitz continuous generative model and for a more specific generative model, based on coefficient decay rate bounds. We examine three high-dimensional problems and show that, for all three examples, the generative model approach outperforms sparsity promoting methods at small sample sizes.

stat.ML↗

arXiv2021

Topology Optimization under Microscale Uncertainty using Stochastic Gradients

This paper considers the design of structures made of engineered materials, accounting for uncertainty in material properties. We present a topology optimization approach that optimizes the structural shape and topology at the macroscale assuming design-independent uncertain microstructures. The structural geometry at the macroscale is described by an explicit level set approach, and the macroscopic structural response is predicted by the eXtended Finite Element Method (XFEM). We describe the microscopic layout by either an analytic geometric model with uncertain parameters or a level cut from a Gaussian random field. The macroscale properties of the microstructured material are predicted by homogenization. Considering the large number of possible microscale configurations, one of the main challenges of solving such topology optimization problems is the computational cost of estimating the statistical moments of the cost and constraint functions and their gradients with respect to the design variables. Methods for predicting these moments, such as Monte Carlo sampling, and Taylor series and polynomial chaos expansions often require many random samples resulting in an impractical computation. To reduce this cost, we propose an approach wherein, at every design iteration, we only use a small number of microstructure configurations to generate an independent, stochastic approximation of the gradients. These gradients are then used either with a gradient descent algorithm, namely Adam, or the globally convergent method of moving asymptotes (GCMMA). Three numerical examples from structural mechanics are used to show that the proposed approach provides a computationally efficient way for macroscale topology optimization in the presence of microstructural uncertainty and enables the designers to consider a new class of problems that are out of reach today with conventional tools.

math.OC↗

arXiv2021

Reliability-based Topology Optimization using Stochastic Gradients

This paper addresses the computational challenges in reliability-based topology optimization (RBTO) of structures associated with the estimation of statistics of the objective and constraints using standard sampling methods, and overcomes the accuracy issues of traditional methods that rely on approximating the limit state function. Herein, we present a stochastic gradient-based approach, where we estimate the probability of failure at every few iterations using an efficient sampling strategy. To estimate the gradients of the failure probability with respect to the design parameters, we apply Bayes' rule wherein we assume a parametric exponential model for the probability density function of the design parameters conditioned on the failure. The design parameters as well as the parameters of this probability density function are updated using a stochastic gradient descent approach requiring only a small, e.g., O(1), number of random samples per iteration, thus leading to a considerable reduction of the computational cost as compared to standard RBTO techniques. We illustrate the proposed approach with a benchmark example that has an analytical solution as well as two widely used problems in structural topology optimization. These examples illustrate the efficacy of the approach in producing reliable designs.

math.OC↗

arXiv2021

Neural Network Training Using $\ell_1$-Regularization and Bi-fidelity Data

With the capability of accurately representing a functional relationship between the inputs of a physical system's model and output quantities of interest, neural networks have become popular for surrogate modeling in scientific applications. However, as these networks are over-parameterized, their training often requires a large amount of data. To prevent overfitting and improve generalization error, regularization based on, e.g., $\ell_1$- and $\ell_2$-norms of the parameters is applied. Similarly, multiple connections of the network may be pruned to increase sparsity in the network parameters. In this paper, we explore the effects of sparsity promoting $\ell_1$-regularization on training neural networks when only a small training dataset from a high-fidelity model is available. As opposed to standard $\ell_1$-regularization that is known to be inadequate, we consider two variants of $\ell_1$-regularization informed by the parameters of an identical network trained using data from lower-fidelity models of the problem at hand. These bi-fidelity strategies are generalizations of transfer learning of neural networks that uses the parameters learned from a large low-fidelity dataset to efficiently train networks for a small high-fidelity dataset. We also compare the bi-fidelity strategies with two $\ell_1$-regularization methods that only use the high-fidelity dataset. Three numerical examples for propagating uncertainty through physical systems are used to show that the proposed bi-fidelity $\ell_1$-regularization strategies produce errors that are one order of magnitude smaller than those of networks trained only using datasets from the high-fidelity models.

stat.ML↗

arXiv2021

A stochastic subspace approach to gradient-free optimization in high dimensions

We present a stochastic descent algorithm for unconstrained optimization that is particularly efficient when the objective function is slow to evaluate and gradients are not easily obtained, as in some PDE-constrained optimization and machine learning problems. The algorithm maps the gradient onto a low-dimensional random subspace of dimension $\ell$ at each iteration, similar to coordinate descent but without restricting directional derivatives to be along the axes. Without requiring a full gradient, this mapping can be performed by computing $\ell$ directional derivatives (e.g., via forward-mode automatic differentiation). We give proofs for convergence in expectation under various convexity assumptions as well as probabilistic convergence results under strong-convexity. Our method extends the well-known Gaussian smoothing technique to descent in subspaces of dimension greater than one, opening the doors to new analysis of Gaussian smoothing when more than one directional derivative is used at each iteration. We also provide a finite-dimensional variant of a special case of the Johnson-Lindenstrauss lemma. Experimentally, we show that our method compares favorably to coordinate descent, Gaussian smoothing, gradient descent and BFGS (when gradients are calculated via forward-mode automatic differentiation) on problems from the machine learning and shape optimization literature.

math.OC↗

arXiv2021

Deterministic matrix sketches for low-rank compression of high-dimensional simulation data

Matrices arising in scientific applications frequently admit linear low-rank approximations due to smoothness in the physical and/or temporal domain of the problem. In large-scale problems, computing an optimal low-rank approximation can be prohibitively expensive. Matrix sketching addresses this by reducing the input matrix to a smaller, but representative matrix via a low-dimensional linear embedding. If the sketch matrix produced by the embedding captures sufficient geometric properties of the original matrix, then a near-optimal approximation may be obtained. Much of the work done in matrix sketching has centered on random projection. Alternatively, in this work, deterministic matrix sketches which generate coarse representations, compatible with the corresponding PDE solve, are considered in the computation of the singular value decomposition and matrix interpolative decomposition. The deterministic sketching approaches in this work have many advantages over randomized sketches. Broadly, randomized sketches are data-agnostic, whereas the proposed sketching methods exploit structures within data generated in complex PDE systems. These deterministic sketches are often faster, require access to a small fraction of the input matrix, and do not need to be explicitly constructed. A novel single-pass, i.e., requiring one read over the input, power iteration algorithm is also presented. The power iteration method is particularly effective in improving low-rank approximations when the singular value decay of data is slow. Finally, theoretical error bounds and estimates, as well as numerical results across three application problems, are provided.

math.NA↗

arXiv2021

Rapid Aerodynamic Shape Optimization Under Parametric and Turbulence Model Uncertainty: A Stochastic Gradient Approach

Aerodynamic optimization is ubiquitous in the design of most engineering systems interacting with fluids. A common approach is to optimize a performance function defined by a choice of an aerodynamic model, e.g., turbulence RANS model, and at nominal operating conditions. Practical experience indicates that such a deterministic approach may result in considerably sub-optimal designs when the adopted aerodynamic model does not lead to accurate flow predictions or when the actual operating conditions differ from those considered in the design. One approach to address this shortcoming is to consider an average or robust design, wherein the statistical moments of the performance function, given the uncertainty in the operating conditions and the aerodynamic model, is optimized. However, when the number of uncertain inputs is large or the performance function exhibits significant variability, an accurate evaluation of these moments may require a large number of forward and/or adjoint solves, at each iteration of a gradient-based scheme. This, in turn, renders the design computationally expensive, if not infeasible. To tackle this difficulty, we consider a variant of the stochastic gradient descent method where, in each optimization iteration, a stochastic approximation of the objective, constraints, and their gradients are generated. This is done via a small number of forward/adjoint solves corresponding to random selections of the uncertain parameters and aerodynamic model. The methodology is applied to the robust optimization of the standard NACA-0012 subject to parametric and turbulence model uncertainty. With a cost that is a small factor larger than that of the deterministic approach, the stochastic gradient approach significantly improves the performance (mean and variance) of the aerodynamic design for a wide range of operating conditions and turbulence models.

math.OC↗