arXiv subjects

Kevin Carlberg

Publications and source records attributed to Kevin Carlberg.

At least 19 recordsLinked to original sources

arXiv2025

DigiData: Training and Evaluating General-Purpose Mobile Control Agents

AI agents capable of controlling user interfaces have the potential to transform human interaction with digital devices. To accelerate this transformation, two fundamental building blocks are essential: high-quality datasets that enable agents to achieve complex and human-relevant goals, and robust evaluation methods that allow researchers and practitioners to rapidly enhance agent performance. In this paper, we introduce DigiData, a large-scale, high-quality, diverse, multi-modal dataset designed for training mobile control agents. Unlike existing datasets, which derive goals from unstructured interactions, DigiData is meticulously constructed through comprehensive exploration of app features, resulting in greater diversity and higher goal complexity. Additionally, we present DigiData-Bench, a benchmark for evaluating mobile control agents on real-world complex tasks. We demonstrate that the commonly used step-accuracy metric falls short in reliably assessing mobile control agents and, to address this, we propose dynamic evaluation protocols and AI-powered evaluations as rigorous alternatives for agent assessment. Our contributions aim to significantly advance the development of mobile control agents, paving the way for more intuitive and effective human-device interactions.

cs.AI↗

arXiv2025

Accelerating scientific discovery with the common task framework

Machine learning (ML) and artificial intelligence (AI) algorithms are transforming and empowering the characterization and control of dynamic systems in the engineering, physical, and biological sciences. These emerging modeling paradigms require comparative metrics to evaluate a diverse set of scientific objectives, including forecasting, state reconstruction, generalization, and control, while also considering limited data scenarios and noisy measurements. We introduce a common task framework (CTF) for science and engineering, which features a growing collection of challenge data sets with a diverse set of practical and common objectives. The CTF is a critically enabling technology that has contributed to the rapid advance of ML/AI algorithms in traditional applications such as speech recognition, language processing, and computer vision. There is a critical need for the objective metrics of a CTF to compare the diverse algorithms being rapidly developed and deployed in practice today across science and engineering.

cs.LG↗

arXiv2025

Benchmarking Egocentric Multimodal Goal Inference for Assistive Wearable Agents

There has been a surge of interest in assistive wearable agents: agents embodied in wearable form factors (e.g., smart glasses) who take assistive actions toward a user's goal/query (e.g. "Where did I leave my keys?"). In this work, we consider the important complementary problem of inferring that goal from multi-modal contextual observations. Solving this "goal inference" problem holds the promise of eliminating the effort needed to interact with such an agent. This work focuses on creating WAGIBench, a strong benchmark to measure progress in solving this problem using vision-language models (VLMs). Given the limited prior work in this area, we collected a novel dataset comprising 29 hours of multimodal data from 348 participants across 3,477 recordings, featuring ground-truth goals alongside accompanying visual, audio, digital, and longitudinal contextual observations. We validate that human performance exceeds model performance, achieving 93% multiple-choice accuracy compared with 84% for the best-performing VLM. Generative benchmark results that evaluate several families of modern vision-language models show that larger models perform significantly better on the task, yet remain far from practical usefulness, as they produce relevant goals only 55% of the time. Through a modality ablation, we show that models benefit from extra information in relevant modalities with minimal performance degradation from irrelevant modalities.

cs.CV↗

arXiv2024

Accelerating Look-ahead in Bayesian Optimization: Multilevel Monte Carlo is All you Need

We leverage multilevel Monte Carlo (MLMC) to improve the performance of multi-step look-ahead Bayesian optimization (BO) methods that involve nested expectations and maximizations. Often these expectations must be computed by Monte Carlo (MC). The complexity rate of naive MC degrades for nested operations, whereas MLMC is capable of achieving the canonical MC convergence rate for this type of problem, independently of dimension and without any smoothness assumptions. Our theoretical study focuses on the approximation improvements for twoand three-step look-ahead acquisition functions, but, as we discuss, the approach is generalizable in various ways, including beyond the context of BO. Our findings are verified numerically and the benefits of MLMC for BO are illustrated on several benchmark examples. Code is available at https://github.com/Shangda-Yang/MLMCBO .

stat.ML↗

arXiv2023

Neural Stress Fields for Reduced-order Elastoplasticity and Fracture

We propose a hybrid neural network and physics framework for reduced-order modeling of elastoplasticity and fracture. State-of-the-art scientific computing models like the Material Point Method (MPM) faithfully simulate large-deformation elastoplasticity and fracture mechanics. However, their long runtime and large memory consumption render them unsuitable for applications constrained by computation time and memory usage, e.g., virtual reality. To overcome these barriers, we propose a reduced-order framework. Our key innovation is training a low-dimensional manifold for the Kirchhoff stress field via an implicit neural representation. This low-dimensional neural stress field (NSF) enables efficient evaluations of stress values and, correspondingly, internal forces at arbitrary spatial locations. In addition, we also train neural deformation and affine fields to build low-dimensional manifolds for the deformation and affine momentum fields. These neural stress, deformation, and affine fields share the same low-dimensional latent space, which uniquely embeds the high-dimensional simulation state. After training, we run new simulations by evolving in this single latent space, which drastically reduces the computation time and memory consumption. Our general continuum-mechanics-based reduced-order framework is applicable to any phenomena governed by the elastodynamics equation. To showcase the versatility of our framework, we simulate a wide range of material behaviors, including elastica, sand, metal, non-Newtonian fluids, fracture, contact, and collision. We demonstrate dimension reduction by up to 100,000X and time savings by up to 10X.

cs.GR↗

arXiv2023

LiCROM: Linear-Subspace Continuous Reduced Order Modeling with Neural Fields

Linear reduced-order modeling (ROM) simplifies complex simulations by approximating the behavior of a system using a simplified kinematic representation. Typically, ROM is trained on input simulations created with a specific spatial discretization, and then serves to accelerate simulations with the same discretization. This discretization-dependence is restrictive. Becoming independent of a specific discretization would provide flexibility to mix and match mesh resolutions, connectivity, and type (tetrahedral, hexahedral) in training data; to accelerate simulations with novel discretizations unseen during training; and to accelerate adaptive simulations that temporally or parametrically change the discretization. We present a flexible, discretization-independent approach to reduced-order modeling. Like traditional ROM, we represent the configuration as a linear combination of displacement fields. Unlike traditional ROM, our displacement fields are continuous maps from every point on the reference domain to a corresponding displacement vector; these maps are represented as implicit neural fields. With linear continuous ROM (LiCROM), our training set can include multiple geometries undergoing multiple loading conditions, independent of their discretization. This opens the door to novel applications of reduced order modeling. We can now accelerate simulations that modify the geometry at runtime, for instance via cutting, hole punching, and even swapping the entire mesh. We can also accelerate simulations of geometries unseen during training. We demonstrate one-shot generalization, training on a single geometry and subsequently simulating various unseen geometries.

cs.GR↗

arXiv2022

Learning a Visually Grounded Memory Assistant

We introduce a novel interface for large scale collection of human memory and assistance. Using the 3D Matterport simulator we create a realistic indoor environments in which we have people perform specific embodied memory tasks that mimic household daily activities. This interface was then deployed on Amazon Mechanical Turk allowing us to test and record human memory, navigation and needs for assistance at a large scale that was previously impossible. Using the interface we collect the `The Visually Grounded Memory Assistant Dataset' which is aimed at developing our understanding of (1) the information people encode during navigation of 3D environments and (2) conditions under which people ask for memory assistance. Additionally we experiment with with predicting when people will ask for assistance using models trained on hand-selected visual and semantic features. This provides an opportunity to build stronger ties between the machine-learning and cognitive-science communities through learned models of human perception, memory, and cognition.

cs.CV↗

arXiv2022

CROM: Continuous Reduced-Order Modeling of PDEs Using Implicit Neural Representations

The long runtime of high-fidelity partial differential equation (PDE) solvers makes them unsuitable for time-critical applications. We propose to accelerate PDE solvers using reduced-order modeling (ROM). Whereas prior ROM approaches reduce the dimensionality of discretized vector fields, our continuous reduced-order modeling (CROM) approach builds a low-dimensional embedding of the continuous vector fields themselves, not their discretization. We represent this reduced manifold using continuously differentiable neural fields, which may train on any and all available numerical solutions of the continuous system, even when they are obtained using diverse methods or discretizations. We validate our approach on an extensive range of PDEs with training data from voxel grids, meshes, and point clouds. Compared to prior discretization-dependent ROM methods, such as linear subspace proper orthogonal decomposition (POD) and nonlinear manifold neural-network-based autoencoders, CROM features higher accuracy, lower memory consumption, dynamically adaptive resolutions, and applicability to any discretization. For equal latent space dimension, CROM exhibits 79$\times$ and 49$\times$ better accuracy, and 39$\times$ and 132$\times$ smaller memory footprint, than POD and autoencoder methods, respectively. Experiments demonstrate 109$\times$ and 89$\times$ wall-clock speedups over unreduced models on CPUs and GPUs, respectively. Videos and codes are available on the project page: https://crom-pde.github.io

cs.LG↗

arXiv2022

Preconditioned Least-Squares Petrov-Galerkin Reduced Order Models

This paper introduces a methodology for improving the accuracy and efficiency of reduced order models (ROMs) constructed using the least-squares Petrov-Galerkin (LSPG) projection method through the introduction of preconditioning. Unlike prior related work, which focuses on preconditioning the linear systems arising within the ROM numerical solution procedure to improve linear solver performance, our approach leverages a preconditioning matrix directly within the LSPG minimization problem. Applying preconditioning in this way can improve ROM accuracy for several reasons. First, preconditioning the LSPG formulation changes the norm defining the residual minimization, which can improve the residual-based stability constant bounding the ROM solution's error. The incorporation of a preconditioner into the LSPG formulation can have the additional effect of scaling the components of the residual being minimized, which can be beneficial for problems with disparate scales. Importantly, we demonstrate that an 'ideal preconditioned' LSPG ROM (a ROM preconditioned with the inverse of the Jacobian of its corresponding full order model, or FOM) emulates projection of the FOM solution increment onto the reduced basis, a lower bound on the ROM solution error for a given reduced basis. By designing preconditioners that approximate the Jacobian inverse, a ROM whose error approaches this lower bound can be obtained. The proposed approach is evaluated in the predictive regime on several mechanical and thermo-mechanical problems within the Albany HPC code. We demonstrate numerically that the introduction of simple Jacobi, Gauss-Seidel and ILU preconditioners into the Proper Orthogonal Decomposition/LSPG formulation reduces significantly the ROM solution error, the reduced Jacobian condition number, the number of nonlinear iterations required to reach convergence, and the wall time.

math.NA↗

arXiv2021

Model reduction for the material point method via an implicit neural representation of the deformation map

This work proposes a model-reduction approach for the material point method on nonlinear manifolds. Our technique approximates the $\textit{kinematics}$ by approximating the deformation map using an implicit neural representation that restricts deformation trajectories to reside on a low-dimensional manifold. By explicitly approximating the deformation map, its spatiotemporal gradients -- in particular the deformation gradient and the velocity -- can be computed via analytical differentiation. In contrast to typical model-reduction techniques that construct a linear or nonlinear manifold to approximate the (finite number of) degrees of freedom characterizing a given spatial discretization, the use of an implicit neural representation enables the proposed method to approximate the $\textit{continuous}$ deformation map. This allows the kinematic approximation to remain agnostic to the discretization. Consequently, the technique supports dynamic discretizations -- including resolution changes -- during the course of the online reduced-order-model simulation. To generate $\textit{dynamics}$ for the generalized coordinates, we propose a family of projection techniques. At each time step, these techniques: (1) Calculate full-space kinematics at quadrature points, (2) Calculate the full-space dynamics for a subset of `sample' material points, and (3) Calculate the reduced-space dynamics by projecting the updated full-space position and velocity onto the low-dimensional manifold and tangent space, respectively. We achieve significant computational speedup via hyper-reduction that ensures all three steps execute on only a small subset of the problem's spatial domain. Large-scale numerical examples with millions of material points illustrate the method's ability to gain an order of magnitude computational-cost saving -- indeed $\textit{real-time simulations}$ -- with negligible errors.

cs.LG↗

arXiv2020

Domain-decomposition least-squares Petrov-Galerkin (DD-LSPG) nonlinear model reduction

A novel domain-decomposition least-squares Petrov-Galerkin (DD-LSPG) model-reduction method applicable to parameterized systems of nonlinear algebraic equations (e.g., arising from discretizing a parameterized partial-differential-equations problem) is proposed. In contrast with previous works, we adopt an algebraically non-overlapping decomposition strategy rather than a spatial-decomposition strategy, which facilitates application to different spatial-discretization schemes. Rather than constructing a low-dimensional subspace for the entire state space in a monolithic fashion, the methodology constructs separate subspaces for the different subdomains/components characterizing the original model. In the offline stage, the method constructs low-dimensional bases for the interior and interface of components. In the online stage, the approach constructs an LSPG ROM for each component and enforces strong or weak compatibility on the 'ports' connecting them. We propose four different ways to construct reduced bases on the interface/ports of subdomains and several ways to enforce compatibility across connecting ports. We derive a posteriori and a priori error bounds for the DD-LSPG solutions. Numerical results performed on nonlinear benchmark problems in heat transfer and fluid dynamics demonstrate that the proposed method performs well in terms of both accuracy and computational cost, with different choices of basis and compatibility constraints yielding different performance profiles.

math.NA↗

arXiv2020

Optimal Assistance for Object-Rearrangement Tasks in Augmented Reality

Augmented-reality (AR) glasses that will have access to onboard sensors and an ability to display relevant information to the user present an opportunity to provide user assistance in quotidian tasks. Many such tasks can be characterized as object-rearrangement tasks. We introduce a novel framework for computing and displaying AR assistance that consists of (1) associating an optimal action sequence with the policy of an embodied agent and (2) presenting this sequence to the user as suggestions in the AR system's heads-up display. The embodied agent comprises a "hybrid" between the AR system and the user, with the AR system's observation space (i.e., sensors) and the user's action space (i.e., task-execution actions); its policy is learned by minimizing the task-completion time. In this initial study, we assume that the AR system's observations include the environment's map and localization of the objects and the user. These choices allow us to formalize the problem of computing AR assistance for any object-rearrangement task as a planning problem, specifically as a capacitated vehicle-routing problem. Further, we introduce a novel AR simulator that can enable web-based evaluation of AR-like assistance and associated at-scale data collection via the Habitat simulator for embodied artificial intelligence. Finally, we perform a study that evaluates user response to the proposed form of AR assistance on a specific quotidian object-rearrangement task, house cleaning, using our proposed AR simulator on mechanical turk. In particular, we study the effect of the proposed AR assistance on users' task performance and sense of agency over a range of task difficulties. Our results indicate that providing users with such assistance improves their overall performance and while users report a negative impact to their agency, they may still prefer the proposed assistance to having no assistance at all.

cs.HC↗

arXiv2020

Deep Conservation: A latent-dynamics model for exact satisfaction of physical conservation laws

This work proposes an approach for latent-dynamics learning that exactly enforces physical conservation laws. The method comprises two steps. First, the method computes a low-dimensional embedding of the high-dimensional dynamical-system state using deep convolutional autoencoders. This defines a low-dimensional nonlinear manifold on which the state is subsequently enforced to evolve. Second, the method defines a latent-dynamics model that associates with the solution to a constrained optimization problem. Here, the objective function is defined as the sum of squares of conservation-law violations over control volumes within a finite-volume discretization of the problem; nonlinear equality constraints explicitly enforce conservation over prescribed subdomains of the problem. Under modest conditions, the resulting dynamics model guarantees that the time-evolution of the latent state exactly satisfies conservation laws over the prescribed subdomains.

physics.comp-ph↗

arXiv2019

The network uncertainty quantification method for propagating uncertainties in component-based systems

This work introduces the network uncertainty quantification (NetUQ) method for performing uncertainty propagation in systems composed of interconnected components. The method assumes the existence of a collection of components, each of which is characterized by exogenous-input random variables, endogenous-input random variables, output random variables, and a local uncertainty-propagation operator that computes output random variables from input random variables. The method assembles the full-system network by connecting components, which is achieved simply by associating endogenous-input random variables for each component with output random variables from other components; no other inter-component compatibility conditions are required. The network uncertainty-propagation problem is: Compute output random variables for all components given all exogenous-input random variables. To solve this problem, the method applies classical relaxation methods (i.e., Jacobi and Gauss--Seidel iteration with Anderson acceleration), which require only black-box evaluations of component uncertainty-propagation operators. Compared with other available methods, this approach is applicable to any network topology, promotes component independence by enabling components to employ tailored uncertainty-propagation operators, supports general functional representations of random variables, and requires no offline preprocessing stage. Also, because the method propagates functional representations of random variables throughout the network (and not, e.g., probability density functions), the joint distribution of any set of random variables throughout the network can be estimated a posteriori in a straightforward manner. We perform supporting convergence and error analysis and execute numerical experiments that demonstrate the weak- and strong-scaling performance of the method.

math.NA↗

arXiv2019

Statistical closure modeling for reduced-order models of stationary systems by the ROMES method

This work proposes a technique for constructing a statistical closure model for reduced-order models (ROMs) applied to stationary systems modeled as parameterized systems of algebraic equations. The proposed technique extends the reduced-order-model error surrogates (ROMES) method to closure modeling. The original ROMES method applied Gaussian-process regression to construct a statistical model that maps cheaply computable error indicators (e.g., residual norm, dual-weighted residuals) to a random variable for either (1) the norm of the state error or (2) the error in a scalar-valued quantity of interest. Rather than target these two types of errors, this work proposes to construct a statistical model for the state error itself; it achieves this by constructing statistical models for the generalized coordinates characterizing both the in-plane error (i.e., the error in the trial subspace) and a low-dimensional approximation of the out-of-plane error. The former can be considered a statistical closure model, as it quantifies the error in the ROM generalized coordinates. Because any quantity of interest can be computed as a functional of the state, the proposed approach enables any quantity-of-interest error to be statistically quantified a posteriori, as the state-error model can be propagated through the associated quantity-of-interest functional. Numerical experiments performed on both linear and nonlinear stationary systems illustrate the ability of the technique (1) to improve (expected) ROM prediction accuracy by an order of magnitude, (2) to statistically quantify the error in arbitrary quantities of interest, and (3) to realize a more cost-effective methodology for reducing the error than a ROM-only approach in the case of nonlinear systems.

math.NA↗

arXiv2018

Model reduction of dynamical systems on nonlinear manifolds using deep convolutional autoencoders

Nearly all model-reduction techniques project the governing equations onto a linear subspace of the original state space. Such subspaces are typically computed using methods such as balanced truncation, rational interpolation, the reduced-basis method, and (balanced) POD. Unfortunately, restricting the state to evolve in a linear subspace imposes a fundamental limitation to the accuracy of the resulting reduced-order model (ROM). In particular, linear-subspace ROMs can be expected to produce low-dimensional models with high accuracy only if the problem admits a fast decaying Kolmogorov $n$-width (e.g., diffusion-dominated problems). Unfortunately, many problems of interest exhibit a slowly decaying Kolmogorov $n$-width (e.g., advection-dominated problems). To address this, we propose a novel framework for projecting dynamical systems onto nonlinear manifolds using minimum-residual formulations at the time-continuous and time-discrete levels; the former leads to manifold Galerkin projection, while the latter leads to manifold least-squares Petrov--Galerkin (LSPG) projection. We perform analyses that provide insight into the relationship between these proposed approaches and classical linear-subspace reduced-order models; we also derive a posteriori discrete-time error bounds for the proposed approaches. In addition, we propose a computationally practical approach for computing the nonlinear manifold, which is based on convolutional autoencoders from deep learning. Finally, we demonstrate the ability of the method to significantly outperform even the optimal linear-subspace ROM on benchmark advection-dominated problems, thereby demonstrating the method's ability to overcome the intrinsic $n$-width limitations of linear subspaces.

math.NA↗

arXiv2017

Conservative model reduction for finite-volume models

This work proposes a method for model reduction of finite-volume models that guarantees the resulting reduced-order model is conservative, thereby preserving the structure intrinsic to finite-volume discretizations. The proposed reduced-order models associate with optimization problems characterized by a minimum-residual objective function and nonlinear equality constraints that explicitly enforce conservation over subdomains. Conservative Galerkin projection arises from formulating this optimization problem at the time-continuous level, while conservative least-squares Petrov--Galerkin (LSPG) projection associates with a time-discrete formulation. We equip these approaches with hyper-reduction techniques in the case of nonlinear flux and source terms, and also provide approaches for handling infeasibility. In addition, we perform analyses that include deriving conditions under which conservative Galerkin and conservative LSPG are equivalent, as well as deriving a posteriori error bounds. Numerical experiments performed on a parameterized quasi-1D Euler equation demonstrate the ability of the proposed method to ensure not only global conservation, but also significantly lower state-space errors than nonconservative reduced-order models such as standard Galerkin and LSPG projection.

math.NA↗

arXiv2017

Structure-preserving model reduction for marginally stable LTI systems

This work proposes a structure-preserving model reduction method for marginally stable linear time-invariant (LTI) systems. In contrast to Lyapunov-stability-based approaches---which ensure the poles of the reduced system remain in the open left-half plane---the proposed method preserves marginal stability by reducing the subsystem with poles on the imaginary axis in a manner that ensures those poles remain purely imaginary. In particular, the proposed method decomposes a marginally stable LTI system into (1) an asymptotically stable subsystem with eigenvalues in the open left-half plane and (2) a pure marginally stable subsystem with a purely imaginary spectrum. We propose a method based on inner-product projection and the Lyapunov inequality to reduce the first subsystem while preserving asymptotic stability. In addition, we demonstrate that the pure marginally stable subsystem is a generalized Hamiltonian system; we then propose a method based on symplectic projection to reduce this subsystem while preserving pure marginal stability. In addition, we propose both inner-product and symplectic balancing methods that balance the operators associated with two quadratic energy functionals while preserving asymptotic and pure marginal stability, respectively. We formulate a geometric perspective that enables a unified comparison of the proposed inner-product and symplectic projection methods. Numerical examples illustrate the ability of the method to reduce the dimensionality of marginally stable LTI systems while retaining accuracy and preserving marginal stability; further, the resulting reduced-order model yields a finite infinite-time energy, which arises from the pure marginally stable subsystem.

math.DS↗