arXiv subjects

Karthik Duraisamy

Publications and source records attributed to Karthik Duraisamy.

At least 37 records · Page 2Linked to original sources

arXiv2025

Easy attention: A simple attention mechanism for temporal predictions with transformers

To improve the robustness of transformer neural networks used for temporal-dynamics prediction of chaotic systems, we propose a novel attention mechanism called easy attention which we demonstrate in time-series reconstruction and prediction. While the standard self attention only makes use of the inner product of queries and keys, it is demonstrated that the keys, queries and softmax are not necessary for obtaining the attention score required to capture long-term dependencies in temporal sequences. Through the singular-value decomposition (SVD) on the softmax attention score, we further observe that self attention compresses the contributions from both queries and keys in the space spanned by the attention score. Therefore, our proposed easy-attention method directly treats the attention scores as learnable parameters. This approach produces excellent results when reconstructing and predicting the temporal dynamics of chaotic systems exhibiting more robustness and less complexity than self attention or the widely-used long short-term memory (LSTM) network. We show the improved performance of the easy-attention method in the Lorenz system, a turbulence shear flow and a model of a nuclear reactor.

cs.LG↗

arXiv2025

Recent developments and research needs in turbulence modeling of hypersonic flows

Hypersonic flow conditions pose exceptional challenges for Reynolds-Averaged Navier-Stokes (RANS) turbulence modeling. Critical phenomena include compressibility effects, shock/turbulent boundary layer interactions, turbulence-chemistry interaction in thermo-chemical non-equilibrium, and ablation-induced surface roughness and blowing effects. This comprehensive review synthesizes recent developments in adapting turbulence models to hypersonic applications, examining approaches ranging from empirical modifications to physics-based reformulations and novel data-driven methodologies. We provide a systematic evaluation of current RANS-based turbulence modeling capabilities, comparing eddy viscosity and Reynolds stress transport formulations in their ability to predict engineering quantities of interest such as separation characteristics and wall heat transfer. Our analysis encompasses the latest experimental and direct numerical simulation datasets for validation, specifically addressing two- and three-dimensional equilibrium turbulent boundary layers and shock/turbulent boundary layer interactions across both smooth and rough surfaces. Key multi-physics considerations including catalysis and ablation phenomena along with the integration of conjugate heat transfer into a RANS solver for efficient design of a thermal protection system are also discussed. We conclude by identifying the critical gaps in the available validation databases and limitations of the existing turbulence models and suggest potential areas for future research to improve the fidelity of turbulence modeling in the hypersonic regime.

physics.flu-dyn↗

arXiv2025

Variational Bayesian Optimal Experimental Design with Normalizing Flows

Bayesian optimal experimental design (OED) seeks experiments that maximize the expected information gain (EIG) in model parameters. Directly estimating the EIG using nested Monte Carlo is computationally expensive and requires an explicit likelihood. Variational OED (vOED), in contrast, estimates a lower bound of the EIG without likelihood evaluations by approximating the posterior distributions with variational forms, and then tightens the bound by optimizing its variational parameters. We introduce the use of normalizing flows (NFs) for representing variational distributions in vOED; we call this approach vOED-NFs. Specifically, we adopt NFs with a conditional invertible neural network architecture built from compositions of coupling layers, and enhanced with a summary network for data dimension reduction. We present Monte Carlo estimators to the lower bound along with gradient expressions to enable a gradient-based simultaneous optimization of the variational parameters and the design variables. The vOED-NFs algorithm is then validated in two benchmark problems, and demonstrated on a partial differential equation-governed application of cathodic electrophoretic deposition and an implicit likelihood case with stochastic modeling of aphid population. The findings suggest that a composition of 4--5 coupling layers is able to achieve lower EIG estimation bias, under a fixed budget of forward model runs, compared to previous approaches. The resulting NFs produce approximate posteriors that agree well with the true posteriors, able to capture non-Gaussian and multi-modal features effectively.

cs.LG↗

arXiv2025

A new dynamic slip approach for wall-modeled Large Eddy Simulations in a Consistent Discontinuous Galerkin Framework

A wall-modeled large eddy simulation approach is proposed in a Discontinuous Galerkin (DG) setting, building on the slip-wall concept of Bae et al. (JFM'19) and the universal scaling relationship by Pradhan and Duraisamy (JFM'23). The effect of the order of the DG approximation is introduced via the length scales in the formulation. The level of under-resolution is represented by a slip Reynolds number and the model attempts to incorporate the effects of the numerical discretization and the subgrid-scale model. The dynamic part of the new model is based on a modified form of Germano identity --performed on the universal scaling parameter-- and is coupled with the dynamic Smagorinsky model. A sharp modal cut-off filter is used as the test filter for the dynamic procedure, and the dynamic model can be easily integrated into any DG solver. Numerical experiments on channel flows show that grid independence of the statistics is achievable and predictions for the mean velocity and Reynolds stress profiles agree well with the direct numerical simulation, even with significant under-resolution. When applied to flows with separation and reattachment, the model also consistently predicts one-point statistics in the reverse flow and post-reattachment regions in good agreement with experiments. The performance of the model in accurately predicting equilibrium and separated flows using significantly under-resolved meshes can be attributed to several aspects that work synergistically: the optimal finite-element projection framework; the interplay of the scale-separation and numerical discretization within the DG framework; and the consistent dynamic procedures for subgrid and wall modeling.

physics.flu-dyn↗

arXiv2025

Stochastic Wavevector Model for Rapidly-Distorted Compressible Turbulence

A stochastic wavevector approach is formulated to accurately represent compressible turbulence subject to rapid deformations. This approach is inspired by the incompressible particle representation model of Kassinos (1995) and preserves the exact nature of compressible Rapid Distortion Theory (RDT). The adoption of a stochastic - rather than the Fourier - perspective simplifies the transformation of statistics to physical space and serves as a starting point for the development of practical turbulence models. We assume small density fluctuations and isentropic flow to obtain a transport equation for the pressure fluctuation. This results in five fewer transport equations compared to the compressible RDT model of Yu and Girimaji (2007). The final formulation is closed in spectral space and only requires numerical approximation for the transformation integrals. The use of Monte Carlo for unit wavevector integration motivates the representation of the moments as stochastic variables. Consistency between the Fourier and stochastic representation is demonstrated by showing equivalency between the evolution equations for the velocity spectrum tensor in both representations. Sample clustering with respect to orientation allows for different techniques to be used for the wavevector magnitude integration. The performance of the stochastic model is evaluated for axially-compressed turbulence, serving as a simplified model for shock-turbulence interaction, and is compared to LIA and DNS. Pure and compressed sheared turbulence at different distortion Mach numbers are also computed and compared to RDT/DNS data. Finally, two additional deformations are applied and compared to solenoidal and pressure-released limits to demonstrate the modeling capability for generic rapid deformations.

physics.flu-dyn↗

arXiv2024

Extracting Koopman Operators for Prediction and Control of Non-linear Dynamics Using Two-stage Learning and Oblique Projections

The Koopman operator framework provides a perspective that non-linear dynamics can be described through the lens of linear operators acting on function spaces. As the framework naturally yields linear embedding models, there have been extensive efforts to utilize it for control, where linear controller designs can be applied to control possibly nonlinear dynamics. However, it is challenging to successfully deploy this modeling procedure in a wide range of applications. In this work, some of the fundamental limitations of linear embedding models are addressed. We show a necessary condition for a linear embedding model to achieve zero modeling error, highlighting a trade-off relation between the model expressivity and a restriction on the model structure to allow the use of linear systems theories for nonlinear dynamics. To achieve good performance despite this trade-off, neural network-based modeling is proposed based on linear embedding with oblique projection, which is derived from a weak formulation of projection-based linear operator learning. We train the proposed model using a two-stage learning procedure, wherein the features and operators are initialized with orthogonal projection, followed by the main training process in which test functions characterizing the oblique projection are learned from data. The first stage achieves an optimality ensured by the orthogonal projection and the second stage improves the generalizability to various tasks by optimizing the model with the oblique projection. We demonstrate the effectiveness of the proposed method over other data-driven modeling methods by providing comprehensive numerical evaluations where four tasks are considered targeting three different systems.

math.OC↗

arXiv2024

Model Predictive Control of Nonlinear Dynamics Using Online Adaptive Koopman Operators

This paper develops a methodology for adaptive data-driven Model Predictive Control (MPC) using Koopman operators. While MPC is ubiquitous in various fields of engineering, the controller performance can deteriorate if the modeling error between the control model and the true dynamics persists, which may often be the case with complex nonlinear dynamics. Adaptive MPC techniques learn models online such that the controller can compensate for the modeling error by incorporating newly available data. We utilize the Koopman operator framework to formulate an adaptive MPC technique that corrects for model discrepancies in a computationally efficient manner by virtue of convex optimization. With the use of neural networks to learn embedding spaces, Koopman operator models enable accurate dynamics modeling. Such complex model forms, however, often lead to unstable online learning. To this end, the proposed method utilizes the soft update of target networks, a technique used in stabilization of model learning in Reinforcement Learning (RL). Also, we provide a discussion on which parameters to be chosen as online updated parameters based on a specific description of linear embedding models. Numerical simulations on a canonical nonlinear dynamical system show that the proposed method performs favorably compared to other data-driven MPC methods while achieving superior computational efficiency through the utilization of Koopman operators.

math.OC↗

arXiv2024

Data-driven Koopman Operator-based Prediction and Control Using Model Averaging

This work presents a data-driven Koopman operator-based modeling method using a model averaging technique. While the Koopman operator has been used for data-driven modeling and control of nonlinear dynamics, it is challenging to accurately reconstruct unknown dynamics from data and perform different decision-making tasks, mainly due to its infinite dimensionality and difficulty of finding invariant subspaces. We utilize ideas from a Bayesian inference-based model averaging technique to devise a data-driven method that first populates multiple Koopman models starting with a feature extraction using neural networks and then computes point estimates of the posterior of predicted variables. Although each model in the ensemble is not likely to be accurate enough for a wide range of operating points or unseen data, the proposed weighted linear embedding model combines the outputs of model ensemble aiming at compensating the modeling error of each model so that the overall performance will be improved.

math.OC↗

arXiv2024

Deterministic and Probabilistic Rounding Error Analysis for Mixed-Precision Arithmetic on Modern Computing Units

Modern computer architectures support low-precision arithmetic, which present opportunities for the adoption of mixed-precision algorithms to achieve high computational throughput and reduce energy consumption. As a growing number of scientific computations leverage specialized hardware accelerators, the risk of rounding errors increases, potentially compromising the reliability of models. This shift towards hardware-optimized, low-precision computations highlights the importance of rounding error analysis to ensure that performance gains do not come at the expense of accuracy, especially in high-stakes scientific applications. In this work, we conduct rounding error analysis on widely used operations such as fused multiply-add (FMA), mixed-precision FMA (MPFMA), and NVIDIA Tensor cores. We present a deterministic and probabilistic approach to quantifying the accumulated rounding errors. Numerical experiments are presented to perform the multiply and accumulate operation (MAC) and matrix-matrix multiplication using Tensor cores with random data. We show that probabilistic bounds produce tighter estimates by nearly an order of magnitude compared to deterministic ones for matrix-matrix multiplication.

stat.CO↗

arXiv2024

Spatially-Aware Diffusion Models with Cross-Attention for Global Field Reconstruction with Sparse Observations

Diffusion models have gained attention for their ability to represent complex distributions and incorporate uncertainty, making them ideal for robust predictions in the presence of noisy or incomplete data. In this study, we develop and enhance score-based diffusion models in field reconstruction tasks, where the goal is to estimate complete spatial fields from partial observations. We introduce a condition encoding approach to construct a tractable mapping mapping between observed and unobserved regions using a learnable integration of sparse observations and interpolated fields as an inductive bias. With refined sensing representations and an unraveled temporal dimension, our method can handle arbitrary moving sensors and effectively reconstruct fields. Furthermore, we conduct a comprehensive benchmark of our approach against a deterministic interpolation-based method across various static and time-dependent PDEs. Our study attempts to addresses the gap in strong baselines for evaluating performance across varying sampling hyperparameters, noise levels, and conditioning methods. Our results show that diffusion models with cross-attention and the proposed conditional encoding generally outperform other methods under noisy conditions, although the deterministic method excels with noiseless data. Additionally, both the diffusion models and the deterministic method surpass the numerical approach in accuracy and computational cost for the steady problem. We also demonstrate the ability of the model to capture possible reconstructions and improve the accuracy of fused results in covariance-based correction tasks using ensemble sampling.

cs.LG↗

arXiv2024

CoCoGen: Physically-Consistent and Conditioned Score-based Generative Models for Forward and Inverse Problems

Recent advances in generative artificial intelligence have had a significant impact on diverse domains spanning computer vision, natural language processing, and drug discovery. This work extends the reach of generative models into physical problem domains, particularly addressing the efficient enforcement of physical laws and conditioning for forward and inverse problems involving partial differential equations (PDEs). Our work introduces two key contributions: firstly, we present an efficient approach to promote consistency with the underlying PDE. By incorporating discretized information into score-based generative models, our method generates samples closely aligned with the true data distribution, showcasing residuals comparable to data generated through conventional PDE solvers, significantly enhancing fidelity. Secondly, we showcase the potential and versatility of score-based generative models in various physics tasks, specifically highlighting surrogate modeling as well as probabilistic field reconstruction and inversion from sparse measurements. A robust foundation is laid by designing unconditional score-based generative models that utilize reversible probability flow ordinary differential equations. Leveraging conditional models that require minimal training, we illustrate their flexibility when combined with a frozen unconditional model. These conditional models generate PDE solutions by incorporating parameters, macroscopic quantities, or partial field measurements as guidance. The results illustrate the inherent flexibility of score-based generative models and explore the synergy between unconditional score-based generative models and the present physically-consistent sampling approach, emphasizing the power and flexibility in solving for and inverting physical fields governed by differential equations, and in other scientific machine learning tasks.

cs.LG↗

arXiv2024

Finite Sample Analysis and Bounds of Generalization Error of Gradient Descent in In-Context Linear Regression

Recent studies show that transformer-based architectures emulate gradient descent during a forward pass, contributing to in-context learning capabilities - an ability where the model adapts to new tasks based on a sequence of prompt examples without being explicitly trained or fine tuned to do so. This work investigates the generalization properties of a single step of gradient descent in the context of linear regression with well-specified models. A random design setting is considered and analytical expressions are derived for the statistical properties and bounds of generalization error in a non-asymptotic (finite sample) setting. These expressions are notable for avoiding arbitrary constants, and thus offer robust quantitative information and scaling relationships. These results are contrasted with those from classical least squares regression (for which analogous finite sample bounds are also derived), shedding light on systematic and noise components, as well as optimal step sizes. Additionally, identities involving high-order products of Gaussian random matrices are presented as a byproduct of the analysis.

math.ST↗

arXiv2024

On the lifting and reconstruction of nonlinear systems with multiple invariant sets

The Koopman operator provides a linear perspective on non-linear dynamics by focusing on the evolution of observables in an invariant subspace. Observables of interest are typically linearly reconstructed from the Koopman eigenfunctions. Despite the broad use of Koopman operators over the past few years, there exist some misconceptions about the applicability of Koopman operators to dynamical systems with more than one disjoint invariant sets (e.g., basins of attractions from isolated fixed points). In this work, we first provide a simple explanation for the mechanism of linear reconstruction-based Koopman operators of nonlinear systems with multiple disjoint invariant sets. Next, we discuss the use of discrete symmetry among such invariant sets to construct Koopman eigenfunctions in a data efficient manner. Finally, several numerical examples are provided to illustrate the benefits of exploiting symmetry for learning the Koopman operator.

math.DS↗

arXiv2024

Enhancing Dynamical System Modeling through Interpretable Machine Learning Augmentations: A Case Study in Cathodic Electrophoretic Deposition

We introduce a comprehensive data-driven framework aimed at enhancing the modeling of physical systems, employing inference techniques and machine learning enhancements. As a demonstrative application, we pursue the modeling of cathodic electrophoretic deposition (EPD), commonly known as e-coating. Our approach illustrates a systematic procedure for enhancing physical models by identifying their limitations through inference on experimental data and introducing adaptable model enhancements to address these shortcomings. We begin by tackling the issue of model parameter identifiability, which reveals aspects of the model that require improvement. To address generalizability , we introduce modifications which also enhance identifiability. However, these modifications do not fully capture essential experimental behaviors. To overcome this limitation, we incorporate interpretable yet flexible augmentations into the baseline model. These augmentations are parameterized by simple fully-connected neural networks (FNNs), and we leverage machine learning tools, particularly Neural Ordinary Differential Equations (Neural ODEs), to learn these augmentations. Our simulations demonstrate that the machine learning-augmented model more accurately captures observed behaviors and improves predictive accuracy. Nevertheless, we contend that while the model updates offer superior performance and capture the relevant physics, we can reduce off-line computational costs by eliminating certain dynamics without compromising accuracy or interpretability in downstream predictions of quantities of interest, particularly film thickness predictions. The entire process outlined here provides a structured approach to leverage data-driven methods. Firstly, it helps us comprehend the root causes of model inaccuracies, and secondly, it offers a principled method for enhancing model performance.

physics.comp-ph↗

arXiv2023

Estimating Global Identifiability Using Conditional Mutual Information in a Bayesian Framework

A novel information-theoretic approach is proposed to assess the global practical identifiability of Bayesian statistical models. Based on the concept of conditional mutual information, an estimate of information gained for each model parameter is used to quantify the identifiability with practical considerations. No assumptions are made about the structure of the statistical model or the prior distribution while constructing the estimator. The estimator has the following notable advantages: first, no controlled experiment or data is required to conduct the practical identifiability analysis; second, unlike popular variance-based global sensitivity analysis methods, different forms of uncertainties, such as model-form, parameter, or measurement can be taken into account; third, the identifiability analysis is global, and therefore independent of a realization of the parameters. If an individual parameter has low identifiability, it can belong to an identifiable subset such that parameters within the subset have a functional relationship and thus have a combined effect on the statistical model. The practical identifiability framework is extended to highlight the dependencies between parameter pairs that emerge a posteriori to find identifiable parameter subsets. The applicability of the proposed approach is demonstrated using a linear Gaussian model and a non-linear methane-air reduced kinetics model. It is shown that by examining the information gained for each model parameter along with its dependencies with other parameters, a subset of parameters that can be estimated with high posterior certainty can be found.

stat.ME↗

arXiv2023

Predictive Reduced Order Modeling of Chaotic Multi-scale Problems Using Adaptively Sampled Projections

An adaptive projection-based reduced-order model (ROM) formulation is presented for model-order reduction of problems featuring chaotic and convection-dominant physics. An efficient method is formulated to adapt the basis at every time-step of the on-line execution to account for the unresolved dynamics. The adaptive ROM is formulated in a Least-Squares setting using a variable transformation to promote stability and robustness. An efficient strategy is developed to incorporate non-local information in the basis adaptation, significantly enhancing the predictive capabilities of the resulting ROMs. A detailed analysis of the computational complexity is presented, and validated. The adaptive ROM formulation is shown to require negligible offline training and naturally enables both future-state and parametric predictions. The formulation is evaluated on representative reacting flow benchmark problems, demonstrating that the ROMs are capable of providing efficient and accurate predictions including those involving significant changes in dynamics due to parametric variations, and transient phenomena. A key contribution of this work is the development and demonstration of a comprehensive ROM formulation that targets predictive capability in chaotic, multi-scale, and transport-dominated problems.

physics.comp-ph↗

arXiv2023

Data-driven Balanced Truncation for Predictive Model Order Reduction of Aeroacoustic Response

Rapid prediction of the aeroacoustic response is a key component in the design of aircraft and turbomachinery. While it is possible to achieve accurate predictions using direct solution of the compressible Navier-Stokes equations, applications of such solvers is not feasible in design optimization due to the high cost of resolving wave phenomena in an Eulerian setting. In this work, we propose a technique for highly accelerated predictions of aeroacoustic response using a data-driven model reduction approach based on the eigensystem realization algorithm (ERA), as a non-intrusive balanced truncation method. Specifically, we create and compare ERA ROMs based on the training data generated by solving the linearized and nonlinear Euler equations with Gaussian pulse inputs, and use them for prediction of the aeroacoustic response of an airfoil subject to different types of gust loading. The results show that both ROMs are in good agreement with the full-order model (FOM) solution in a purely predictive setting, while achieving orders of magnitude reduction in the online computation time. Using ERA for prediction of the acoustic response requires activating each input channel separately in the FOM for training ROMs, and operating on a large Hankel matrix, that can become computationally infeasible. We address this bottleneck in two steps: first, we propose a multi-fidelity gappy POD method to identify the most impactful input channels based on a coarser grid. Therefore, we reduce the computation cost on the FOM and ROM levels as we build the Markov sequence by querying the high-resolution FOM only for the input channels identified by gappy POD. Second, we use tangential interpolation at the ROM level to reduce the size of the Hankel matrix. The proposed methods enable application of ERA for highly accurate online acoustic response prediction and reduce the offline computation cost of ROMs.

physics.flu-dyn↗

arXiv2023

Control-aware Learning of Koopman Embedding Models

A learning method is proposed for Koopman operator-based models with the goal of improving closed-loop control behavior. A neural network-based approach is used to discover a space of observables in which nonlinear dynamics is linearly embedded. While accurate state predictions can be expected with the use of such complex state-to-observable maps, undesirable side-effects may be introduced when the model is deployed in a closed-loop environment. This is because of modeling or residual error in the linear embedding process, which can manifest itself in a different manner compared to the state prediction. To this end, a technique is proposed to refine the originally trained model with the goal of improving the closed-loop behavior of the model while retaining the state-prediction accuracy obtained in the initial learning. Finally, a simple data sampling strategy is proposed to use inputs deterministically sampled from continuous functions, leading to additional improvements in the controller performance for nonlinear dynamical systems. Several numerical examples are provided to show the efficacy of the proposed method.

math.OC↗