arXiv subjects

Tapio Schneider

Publications and source records attributed to Tapio Schneider.

At least 19 recordsLinked to original sources

arXiv2026

Improving precipitation forecasts in an AI weather model using observational data

Artificial intelligence weather prediction systems now surpass state-of-the-art physical models for medium-range forecasting. However, because these models are trained almost exclusively on historical climate reanalyses, they inherit pervasive structural biases, particularly for precipitation. Here we fine-tune a global graph-transformer architecture directly on high-resolution, satellite-derived precipitation observations. The resulting model reduces global medium-range probabilistic forecasting errors by up to 19% and improves extreme rainfall prediction accuracy by 57% over current operational models. It additionally demonstrates superior skill for tropical storms and drizzle events, though a physics-based operational model remains more reliable for the heaviest precipitation events. Our results demonstrate that incorporating observation-based data directly into training can substantially improve precipitation forecasts.

physics.ao-ph↗

arXiv2026

A scaling theory for the macroturbulence of weakly supercritical planetary atmospheres

A theory for the general circulation of the atmosphere must be based on a theory of its macroturbulence. A central component is the near-surface eddy heat flux, which is closely tied to the mass transport of the surface branch of the circulation. Existing scaling theories are largely formulated within the two-layer quasi-geostrophic framework, which assumes supercritical states and an inverse cascade of kinetic energy---conditions that are frequently not satisfied in planetary atmospheres, including Earth's. Nevertheless, such theories have shown some empirical success, including that the Rhines scale is associated with the mixing length even in the absence of any inverse energy cascade. Here, we analyze hundreds of idealized, dry general circulation model simulations spanning wide ranges of rotation rates, meridional temperature gradients, vertical stratification, and seasonality. Most simulations reside in the marginally critical regime where strongly nonlinear scaling theories fail. We propose a new theory that captures both domain-averaged and local eddy heat flux behavior across all simulations. The theory uses scaling arguments for the zonal momentum balance and accounts for kinematic effects on eddy mixing. The scaling depends on a non-dimensional thermal Rossby number and implicitly accounts for the role of nonlinear eddy momentum fluxes across jets. We demonstrate the skill of the proposed scaling in explaining global near-surface temperature distributions across a wide range of climate states in a simple energy balance model. Implications for general circulation theory and extending the scalings to moist atmospheres are discussed.

physics.ao-ph↗

arXiv2026

The Ensemble Kalman Inversion Race

Ensemble Kalman methods were initially developed to solve nonlinear data assimilation problems in oceanography but are now popular in applications far beyond their original use cases. Of particular interest is climate model calibration. As hybrid physics and machine-learning models advance, the number of parameters and complexity of parameterizations in climate models will continue to grow. To fully realize these advances, we must move from laborious hand-tuning to calibration-driven model development in rapid iteration cycles. Thus, robust calibration of these parameters plays an increasingly important role. We focus on learning climate model parameters by minimizing the misfit between modeled and observed climate statistics in an idealized setting. Ensemble Kalman methods are a natural choice for this problem because they are derivative-free, scalable to high dimensions, and robust to noise caused by statistical observations. Given the many variants of ensemble methods proposed, an important question is: Which ensemble Kalman method should be used for climate model calibration? To answer this question, we perform systematic numerical experiments to explore the relative computational efficiencies of several ensemble Kalman methods. The numerical experiments involve statistical observations of Lorenz-type models of increasing complexity, frequently used to represent simplified atmospheric systems, with some featuring neural network parameterizations. For each test problem, several ensemble Kalman methods and a derivative-based method "race" to reach a specified accuracy, and we measure the computational cost required to achieve the desired accuracy. We investigate how prior information and the parameter or data dimensions define the computational costs of the various methods.

physics.data-an↗

arXiv2026

A Next-Generation Snow Albedo Parameterization for Climate Modeling using Constrained Machine Learning

We demonstrate a data-driven parameterization for snow albedo using a constrained neural differential equation that directly predicts a range of snow albedo tendencies from standard snow and meteorological inputs. After training with multi-year in-situ and satellite observations from a wide variety of locations, the scheme effectively reproduces daily albedo evolution across diverse climate zones, with median error under 7.5% (RMSE ~0.05), a 10-30% improvement over established models. Furthermore, the model generalizes to sites not seen during training and scales from coarser grids to point locations. The scheme can easily incorporate new features as observational networks expand, offering an adaptive and computationally lightweight framework for next-generation land and climate models.

physics.app-ph↗

arXiv2026

Agents' Last Exam

Recent AI systems have achieved strong results on a wide range of benchmarks, yet these gains have not translated into economically meaningful deployment across many professional domains. We argue that this gap is largely an evaluation problem: widely used benchmarks lack sustained performance measurement on real and economically valuable workflows. This paper introduces Agents' Last Exam (ALE), a benchmark designed to evaluate AI agents on long horizon, economically valuable, real world tasks with verifiable outcomes. Developed in collaboration with 250+ industry experts, ALE covers non-physical industries defined with reference to O*NET / SOC 2018 (the U.S. federal occupational taxonomy). It is organized around a task taxonomy with 55 sub fields grouped into 13 industry clusters covering 1K+ tasks. Current results show that the hardest tier remains far from saturated: across mainstream harness and backbone configurations, the average full pass rate is below 1%. ALE is designed as a living benchmark: its task pool grows continuously as new workflows and industries are onboarded. More broadly, ALE is intended not merely as another leaderboard, but as an instrument for closing the gap between benchmark success and GDP relevant impact.

cs.AI↗

arXiv2026

Probabilistic Seasonal Streamflow Forecasting Across California's Sierra Nevada Watersheds with Agentic AI

Accurate seasonal runoff forecasts are critical for managing California's reservoirs and water supply for millions of its residents. Winter snow accumulation provides a strong source of predictability of snowmelt-based runoff in the spring and summer months, but progressive hydroclimatic changes in the Sierra Nevada are altering its timing and volume. These changes reduce the skill of statistical forecasts trained on historical data, highlighting the need for improved forecasting systems that can capture the changing dynamics of snowmelt. Here we demonstrate that a collaborative workflow between an agentic AI assistant and an automated code-mutation system, both powered by large language models, can accelerate the development of competitive seasonal runoff forecasting systems. In our framework, the AI agent discovers relevant datasets, synthesizes domain knowledge from prior forecasting competitions and the scientific literature, and explores the space of model architectures, while the code-mutation system refines each of the solutions explored by the agent through Monte Carlo Tree Search over the code space. The resulting system forecasts monthly Full Natural Flow (FNF) at 1- to 6-month lead times across 23 Sierra Nevada watersheds using an adaptive ensemble of three XGBoost quantile regression sub-models with physics-informed feature engineering. Evaluated against California's operational Bulletin 120 forecasts over 2021-2025, the agent-evolved model achieves superior skill for early-season cumulative April-July runoff predictions, reducing watershed-averaged quantile forecast error by up to 29%, and offering a new paradigm for AI-driven scientific model development in the geosciences.

physics.ao-ph↗

arXiv2026

Regional climate risk assessment from climate models using probabilistic machine learning

Effective climate risk assessment is hindered by the resolution gap between coarse global climate models and the fine-scale information needed for regional decisions. We introduce GenFocal, an AI framework that generates statistically accurate, fine-scale weather from coarse climate projections, without requiring paired simulated and observed events during training. GenFocal synthesizes complex and long-lived hazards, such as heat waves and tropical cyclones, even when they are not well represented in the coarse climate projections. It also samples high-impact, rare events more accurately than leading methods. By translating large-scale climate projections into actionable, localized information, GenFocal provides a powerful new paradigm to improve climate adaptation and resilience strategies.

cs.LG↗

arXiv2025

A Physics-Constrained Neural Differential Equation Framework for Data-Driven Snowpack Simulation

This paper presents a physics-constrained neural differential equation framework for parameterization, and employs it to model the time evolution of seasonal snow depth given hydrometeorological forcings. When trained on data from multiple SNOTEL sites, the parameterization predicts daily snow depth with under 9% median error and Nash Sutcliffe Efficiencies over 0.94 across a wide variety of snow climates. The parameterization also generalizes to new sites not seen during training, which is not often true for calibrated snow models. Requiring the parameterization to predict snow water equivalent in addition to snow depth only increases error to ~12%. The structure of the approach guarantees the satisfaction of physical constraints, enables these constraints during model training, and allows modeling at different temporal resolutions without additional retraining of the parameterization. These benefits hold potential in climate modeling, and could extend to other dynamical systems with physical constraints.

cs.LG↗

arXiv2025

Reconciling Jupiter's Vertical Motions with the Observed Cloud Structure in the Upper Troposphere

The eddy fluxes of angular momentum in Jupiter's upper troposphere are known to converge in prograde jets and diverge in retrograde jets. Away from the equator, this implies convergence of the Eulerian mean meridional flow in zones (anticyclonic shear) and divergence in belts (cyclonic shear). It indicates lower-tropospheric downwelling in zones and upwelling in belts because the mean meridional circulation almost certainly closes at depth. Yet the observed banded structure of Jupiter's clouds and hazes suggests that there is upwelling in the brighter zones and downwelling in the darker belts. Here, we show that this apparent contradiction can be resolved by considering not the Eulerian but the transformed Eulerian mean circulation, which includes a Stokes drift owing to eddies and is a better approximation of the Lagrangian mean transport of tracers such as ammonia. The potential vorticity structure inferred from observations paired with mixing length arguments suggests that there is transformed Eulerian mean upwelling in zones and downwelling in belts. Simulations with a global circulation model of Jupiter's upper atmosphere demonstrate the plausibility of these inferences and allow us to speculate on the band structure at deeper levels.

astro-ph.EP↗

arXiv2025

High-resolution simulations unravel intensification mechanisms of pyrocumulonimbus clouds

Pyrocumulonimbus (pyroCb) firestorms -- wildfire-generated thunderstorms -- can trigger rapid fire spread. However, the multi-physics nature of pyroCb has made their core mechanisms inaccessible to direct observation and previous simulation and prediction efforts. We introduce a new simulation capability with the first high-resolution, fully coupled simulations of a pyroCb, allowing us to unravel its life cycle governed by two opposing mechanisms. We show fuel moisture is an energy sink that attenuates fire intensity rather than fueling clouds, resolving a long-standing debate. Conversely, we identify the driver of rapid intensification: the Self-Amplifying Fire-Induced Recirculation (SAFIR) mechanism, where precipitation-induced downdrafts intensify the parent fire under weak winds. This work provides a new mechanistic framework for pyroCb prediction and demonstrates a transformative computational approach for previously intractable problems in environmental science.

physics.ao-ph↗

arXiv2025

Online learning of eddy-viscosity and backscattering closures for geophysical turbulence using ensemble Kalman inversion

Different approaches to using data-driven methods for subgrid-scale closure modeling have emerged recently. Most of these approaches are data-hungry, and lack interpretability and out-of-distribution generalizability. Here, we use {online} learning to address parametric uncertainty of well-known physics-based large-eddy simulation (LES) closures: the Smagorinsky (Smag) and Leith eddy-viscosity models (1 free parameter) and the Jansen-Held (JH) backscattering model (2 free parameters). For 8 cases of 2D geophysical turbulence, optimal parameters are estimated, using ensemble Kalman inversion (EKI), such that for each case, the LES' energy spectrum matches that of direct numerical simulation (DNS). Only a small training dataset is needed to calculate the DNS spectra (i.e., the approach is {data-efficient}). We find the optimized parameter(s) of each closure to be constant across broad flow regimes that differ in dominant length scales, eddy/jet structures, and dynamics, suggesting that these closures are {generalizable}. In a-posteriori tests based on the enstrophy spectra and probability density functions (PDFs) of vorticity, LES with optimized closures outperform the baselines (LES with standard Smag, dynamic Smag or Leith), particularly at the tails of the PDFs (extreme events). In a-priori tests, the optimized JH significantly outperforms the baselines and optimized Smag and Leith in terms of interscale enstrophy and energy transfers (still, optimized Smag noticeably outperforms standard Smag). The results show the promise of combining advances in physics-based modeling (e.g., JH) and data-driven modeling (e.g., {online} learning with EKI) to develop data-efficient frameworks for accurate, interpretable, and generalizable closures.

physics.flu-dyn↗

arXiv2024

Toward Routing River Water in Land Surface Models with Recurrent Neural Networks

Machine learning is playing an increasing role in hydrology, supplementing or replacing physics-based models. One notable example is the use of recurrent neural networks (RNNs) for forecasting streamflow given observed precipitation and geographic characteristics. Training of such a model over the continental United States (CONUS) has demonstrated that a single set of model parameters can be used across independent catchments, and that RNNs can outperform physics-based models. In this work, we take a next step and study the performance of RNNs for river routing in land surface models (LSMs). Instead of observed precipitation, the LSM-RNN uses instantaneous runoff calculated from physics-based models as an input. We train the model with data from river basins spanning the globe and test it using historical streamflow measurements. The model demonstrates skill at generalization across basins (predicting streamflow in catchments not used in training) and across time (predicting streamflow during years not used in training). We compare the predictions from the LSM-RNN to an existing physics-based model calibrated with a similar dataset and find that the LSM-RNN outperforms the physics-based model: a gain in median NSE from 0.56 to 0.64 (time-split experiment) and from 0.30 to 0.34 (basin-split experiment). Our results show that RNNs are effective for global streamflow prediction from runoff inputs and motivate the development of complete routing models that can capture nested sub-basis connections.

physics.comp-ph↗

arXiv2024

Dynamical-generative downscaling of climate model ensembles

Regional high-resolution climate projections are crucial for many applications, such as agriculture, hydrology, and natural hazard risk assessment. Dynamical downscaling, the state-of-the-art method to produce localized future climate information, involves running a regional climate model (RCM) driven by an Earth System Model (ESM), but it is too computationally expensive to apply to large climate projection ensembles. We propose a novel approach combining dynamical downscaling with generative artificial intelligence to reduce the cost and improve the uncertainty estimates of downscaled climate projections. In our framework, an RCM dynamically downscales ESM output to an intermediate resolution, followed by a generative diffusion model that further refines the resolution to the target scale. This approach leverages the generalizability of physics-based models and the sampling efficiency of diffusion models, enabling the downscaling of large multi-model ensembles. We evaluate our method against dynamically-downscaled climate projections from the CMIP6 ensemble. Our results demonstrate its ability to provide more accurate uncertainty bounds on future regional climate than alternatives such as dynamical downscaling of smaller ensembles, or traditional empirical statistical downscaling methods. We also show that dynamical-generative downscaling results in significantly lower errors than bias correction and spatial disaggregation (BCSD), and captures more accurately the spectra and multivariate correlations of meteorological fields. These characteristics make the dynamical-generative framework a flexible, accurate, and efficient way to downscale large ensembles of climate projections, currently out of reach for pure dynamical downscaling.

physics.ao-ph↗

arXiv2024

Learning About Structural Errors in Models of Complex Dynamical Systems

Complex dynamical systems are notoriously difficult to model because some degrees of freedom (e.g., small scales) may be computationally unresolvable or are incompletely understood, yet they are dynamically important. For example, the small scales of cloud dynamics and droplet formation are crucial for controlling climate, yet are unresolvable in global climate models. Semi-empirical closure models for the effects of unresolved degrees of freedom often exist and encode important domain-specific knowledge. Building on such closure models and correcting them through learning the structural errors can be an effective way of fusing data with domain knowledge. Here we describe a general approach, principles, and algorithms for learning about structural errors. Key to our approach is to include structural error models inside the models of complex systems, for example, in closure models for unresolved scales. The structural errors then map, usually nonlinearly, to observable data. As a result, however, mismatches between model output and data are only indirectly informative about structural errors, due to a lack of labeled pairs of inputs and outputs of structural error models. Additionally, derivatives of the model may not exist or be readily available. We discuss how structural error models can be learned from indirect data with derivative-free Kalman inversion algorithms and variants, how sparsity constraints enforce a "do no harm" principle, and various ways of modeling structural errors. We also discuss the merits of using non-local and/or stochastic error models. In addition, we demonstrate how data assimilation techniques can assist the learning about structural errors in non-ergodic systems. The concepts and algorithms are illustrated in two numerical examples based on the Lorenz-96 system and a human glucose-insulin model.

physics.comp-ph↗

arXiv2023

Accelerating large-eddy simulations of clouds with Tensor Processing Units

Clouds, especially low clouds, are crucial for regulating Earth's energy balance and mediating the response of the climate system to changes in greenhouse gas concentrations. Despite their importance for climate, they remain relatively poorly understood and are inaccurately represented in climate models. A principal reason is that the high computational expense of simulating them with large-eddy simulations (LES) has inhibited broad and systematic numerical experimentation and the generation of large datasets for training parametrization schemes for climate models. Here we demonstrate LES of low clouds on Tensor Processing Units (TPUs), application-specific integrated circuits that were originally developed for machine learning applications. We show that TPUs in conjunction with tailored software implementations can be used to simulate computationally challenging stratocumulus clouds in conditions observed during the Dynamics and Chemistry of Marine Stratocumulus (DYCOMS) field study. The TPU-based LES code successfully reproduces clouds during DYCOMS and opens up the large computational resources available on TPUs to cloud simulations. The code enables unprecedented weak and strong scaling of LES, making it possible, for example, to simulate stratocumulus with $10\times$ speedup over real-time evolution in domains with a $34.7~\mathrm{km} \times 53.8~\mathrm{km}$ horizontal cross section. The results open up new avenues for computational experiments and for substantially enlarging the sample of LES available to train parameterizations of low clouds.

physics.ao-ph↗

arXiv2022

Ensemble-Based Experimental Design for Targeting Data Acquisition to Inform Climate Models

Data required to calibrate uncertain GCM parameterizations are often only available in limited regions or time periods, for example, observational data from field campaigns, or data generated in local high-resolution simulations. This raises the question of where and when to acquire additional data to be maximally informative about parameterizations in a GCM. Here we construct a new ensemble-based parallel algorithm to automatically target data acquisition to regions and times that maximize the uncertainty reduction, or information gain, about GCM parameters. The algorithm uses a Bayesian framework that exploits a quantified distribution of GCM parameters as a measure of uncertainty. This distribution is informed by time-averaged climate statistics restricted to local regions and times. The algorithm is embedded in the recently developed calibrate-emulate-sample (CES) framework, which performs efficient model calibration and uncertainty quantification with only $\mathcal{O}(10^2)$ model evaluations, compared with $\mathcal{O}(10^5)$ evaluations typically needed for traditional approaches to Bayesian calibration. We demonstrate the algorithm with an idealized GCM, with which we generate surrogates of local data. In this perfect-model setting, we calibrate parameters and quantify uncertainties in a quasi-equilibrium convection scheme in the GCM. We consider targeted data that are (i) localized in space for statistically stationary simulations, and (ii) localized in space and time for seasonally varying simulations. In these proof-of-concept applications, the calculated information gain reflects the reduction in parametric uncertainty obtained from Bayesian inference when harnessing a targeted sample of data. The largest information gain typically, but not always, results from regions near the intertropical convergence zone (ITCZ).

stat.AP↗

arXiv2022

Epidemic Management and Control Through Risk-Dependent Individual Contact Interventions

Testing, contact tracing, and isolation (TTI) is an epidemic management and control approach that is difficult to implement at scale because it relies on manual tracing of contacts. Exposure notification apps have been developed to digitally scale up TTI by harnessing contact data obtained from mobile devices; however, exposure notification apps provide users only with limited binary information when they have been directly exposed to a known infection source. Here we demonstrate a scalable improvement to TTI and exposure notification apps that uses data assimilation (DA) on a contact network. Network DA exploits diverse sources of health data together with the proximity data from mobile devices that exposure notification apps rely upon. It provides users with continuously assessed individual risks of exposure and infection, which can form the basis for targeting individual contact interventions. Simulations of the early COVID-19 epidemic in New York City prove the concepts. In the simulations, network DA identifies up to a factor 2 more infections than contact tracing when both harness the same contact data and diagnostic test data. This remains true even when only a relatively small fraction of the population uses network DA. When a sufficiently large fraction of the population ($\gtrsim 75\%$) uses network DA and complies with individual contact interventions, targeting contact interventions with network DA reduces deaths by up to a factor 4 relative to TTI. Network DA can be implemented by expanding the computational backend of existing exposure notification apps, thus greatly enhancing their capabilities. Implemented at scale, it has the potential to precisely and effectively control future epidemics while minimizing economic disruption.

stat.AP↗

arXiv2022

Iterated Kalman Methodology For Inverse Problems

This paper is focused on the optimization approach to the solution of inverse problems. We introduce a stochastic dynamical system in which the parameter-to-data map is embedded, with the goal of employing techniques from nonlinear Kalman filtering to estimate the parameter given the data. The extended Kalman filter (which we refer to as ExKI in the context of inverse problems) can be effective for some inverse problems approached this way, but is impractical when the forward map is not readily differentiable and is given as a black box, and also for high dimensional parameter spaces because of the need to propagate large covariance matrices. Application of ensemble Kalman filters, for example use of the ensemble Kalman inversion (EKI) algorithm, has emerged as a useful tool which overcomes both of these issues: it is derivative free and works with a low-rank covariance approximation formed from the ensemble. In this paper, we work with the ExKI, EKI, and a variant on EKI which we term unscented Kalman inversion (UKI). The paper contains two main contributions. Firstly, we identify a novel stochastic dynamical system in which the parameter-to-data map is embedded. We present theory in the linear case to show exponential convergence of the mean of the filtering distribution to the solution of a regularized least squares problem. This is in contrast to previous work in which the EKI has been employed where the dynamical system used leads to algebraic convergence to an unregularized problem. Secondly, we show that the application of the UKI to this novel stochastic dynamical system yields improved inversion results, in comparison with the application of EKI to the same novel stochastic dynamical system.

math.NA↗