arXiv subjects

Biwei Dai

Publications and source records attributed to Biwei Dai.

At least 19 recordsLinked to original sources

arXiv2026

The information content of optical intensity interferometry

Optical intensity interferometry (II) can in principle resolve structure on angular scales $\theta\sim \lambda/B$, but it measures only the squared visibility $|{\cal V}|^2$, discarding Fourier phase. We quantify what information can actually be recovered from phase-less $|{\cal V}|^2$ measurements as a function of the aggregate signal-to-noise ratio and array configuration, using score-based diffusion models as non-parametric image priors. We construct an image reconstruction pipeline, and validate it by reconstructing a variety of images, including GRMHD simulations of black hole accretion disks and actual SDO/HMI continuum images of the Sun, from phase-less Fourier amplitudes. Using simulated data, we map the reconstruction fidelity of Sun-like stellar surfaces as a function of noise level, number of $(u,v)$ samples, and maximum baseline $B_{\rm max}$, for surface maps spanning a range of activity levels. We find that fidelity depends on the data only through the total signal to noise ratio (SNR) of the surface features of interest, provided that the $(u,v)$ plane is sampled sufficiently densely. These results show that the SNR levels achieved by next-generation intensity interferometers observing realistic sources are sufficient to image highly complicated stellar surfaces.

astro-ph.IM↗

arXiv2026

FAIR Universe Weak Lensing ML Uncertainty Challenge: Handling Uncertainties and Distribution Shifts for Precision Cosmology

Weak gravitational lensing, the correlated distortion of background galaxy shapes by foreground structures, is a powerful probe of the matter distribution in our universe and allows accurate constraints on the cosmological model. In recent years, high-order statistics and machine learning (ML) techniques have been applied to weak lensing data to extract the nonlinear information beyond traditional two-point analysis. However, these methods typically rely on cosmological simulations, which poses several challenges: simulations are computationally expensive, limiting most realistic setups to a low training data regime; inaccurate modeling of systematics in the simulations create distribution shifts that can bias cosmological parameter constraints; and varying simulation setups across studies make method comparison difficult. To address these difficulties, we present the first weak lensing benchmark dataset with several realistic systematics and launch the FAIR Universe Weak Lensing Machine Learning Uncertainty Challenge. The challenge focuses on measuring the fundamental properties of the universe from weak lensing data with limited training set and potential distribution shifts, while providing a standardized benchmark for rigorous comparison across methods. Organized in two phases, the challenge will bring together the physics and ML communities to advance the methodologies for handling systematic uncertainties, data efficiency, and distribution shifts in weak lensing analysis with ML, ultimately facilitating the deployment of ML approaches into upcoming weak lensing survey analysis.

astro-ph.CO↗

arXiv2026

Competing with AI Scientists: Agent-Driven Approach to Astrophysics Research

We present an agent-driven approach to the construction of parameter inference pipelines for scientific data analysis. Our method leverages a multi-agent system, Cmbagent (the analysis system of the AI scientist Denario), in which specialized agents collaborate to generate research ideas, write and execute code, evaluate results, and iteratively refine the overall pipeline. As a case study, we apply this approach to the FAIR Universe Weak Lensing Uncertainty Challenge, a competition under time constraints focused on robust cosmological parameter inference with realistic observational uncertainties. While the fully autonomous exploration initially did not reach expert-level performance, the integration of human intervention enabled our agent-driven workflow to achieve a first-place result in the challenge. This demonstrates that semi-autonomous agentic systems can compete with, and in some cases surpass, expert solutions. We describe our workflow in detail, including both the autonomous and semi-autonomous exploration by Cmbagent. Our final inference pipeline utilizes parameter-efficient convolutional neural networks, likelihood calibration over a known parameter grid, and multiple regularization techniques. Our results suggest that agent-driven research workflows can provide a scalable framework to rapidly explore and construct pipelines for inference problems.

cs.AI↗

arXiv2025

One latent to fit them all: a unified representation of baryonic feedback on matter distribution

Accurate and parsimonious quantification of baryonic feedback on matter distribution is of crucial importance for understanding both cosmology and galaxy formation from observational data. This is, however, challenging given the large discrepancy among different models of galaxy formation simulations, and their distinct subgrid physics parameterizations. Using 5,072 simulations from 4 different models covering broad ranges in their parameter spaces, we find a unified 2D latent representation. Compared to the simulations and other phenomenological models, our representation is independent of both time and cosmology, much lower-dimensional, and disentangled in its impacts on the matter power spectra. The common latent space facilitates the comparison of parameter spaces of different models and is readily interpretable by correlation with each. The two latent dimensions provide a complementary representation of baryonic effects, linking black hole and supernova feedback to distinct and interpretable impacts on both the matter power spectrum, and field, level. Our approach enables developing robust and economical analytic models for optimal gain of physical information from data, and is generalizable to other fields with significant modeling uncertainty.

astro-ph.CO↗

arXiv2025

Detecting Modeling Bias with Continuous Time Flow Models on Weak Lensing Maps

Simulation-based inference provides a powerful framework for extracting rich information from nonlinear scales in current and upcoming cosmological surveys, and ensuring its robustness requires stringent validation of forward models. In this work, we recast forward model validation as an out-of-distribution (OoD) detection problem within the framework of machine learning (ML)-based simulation-based inference (SBI). We employ probability density as the metric for OoD detection, and compare various density estimation techniques, demonstrating that field-level probability density estimation via continuous time flow models (CTFM) significantly outperforms feature-level approaches that combine scattering transform (ST) or convolutional neural networks (CNN) with normalizing flows (NFs), as well as NF-based field-level estimators, as quantified by the area under the receiver operating characteristic curve (AUROC). Our analysis shows that CTFM not only excels in detecting OoD samples but also provides a robust metric for model selection. Additionally, we verified CTFM maintains consistent efficacy across different cosmologies while mitigating the inductive biases inherent in NF architectures. Although our proof-of-concept study employs simplified forward modeling and noise settings, our framework establishes a promising pathway for identifying unknown systematics in the cosmology datasets.

astro-ph.CO↗

arXiv2024

Rubin ToO 2024: Envisioning the Vera C. Rubin Observatory LSST Target of Opportunity program

The Legacy Survey of Space and Time (LSST) at Vera C. Rubin Observatory is planned to begin in the Fall of 2025. The LSST survey cadence has been designed via a community-driven process regulated by the Survey Cadence Optimization Committee (SCOC), which recommended up to 3% of the observing time to carry out Target of Opportunity (ToO) observations. Experts from the scientific community, Rubin Observatory personnel, and members of the SCOC were brought together to deliver a recommendation for the implementation of the ToO program during a workshop held in March 2024. Four main science cases were identified: gravitational wave multi-messenger astronomy, high energy neutrinos, Galactic supernovae, and small potentially hazardous asteroids possible impactors. Additional science cases were identified and briefly addressed in the documents, including lensed or poorly localized gamma-ray bursts and twilight discoveries. Trigger prioritization, automated response, and detailed strategies were discussed for each science case. This document represents the outcome of the Rubin ToO 2024 workshop, with additional contributions from members of the Rubin Science Collaborations. The implementation of the selection criteria and strategies presented in this document has been endorsed in the SCOC Phase 3 Recommendations document (PSTN-056). Although the ToO program is still to be finalized, this document serves as a baseline plan for ToO observations with the Rubin Observatory.

astro-ph.IM↗

arXiv2024

A comparative study of cosmological constraints from weak lensing using Convolutional Neural Networks

Weak Lensing (WL) surveys are reaching unprecedented depths, enabling the investigation of very small angular scales. At these scales, nonlinear gravitational effects lead to higher-order correlations making the matter distribution highly non-Gaussian. Extracting this information using traditional statistics has proven difficult, and Machine Learning based summary statistics have emerged as a powerful alternative. We explore the capabilities of a discriminative, Convolutional Neural Networks (CNN) based approach, focusing on parameter constraints in the ($\Omega_m$, $\sigma_8$) cosmological parameter space. Leveraging novel training loss functions and network representations on WL mock datasets without baryons, we show that our models achieve $\sim 5$ times stronger constraints than the power spectrum, $\sim 3$ stronger constraints than peak counts, and $\sim 2$ stronger constraints than previous CNN-learned summary statistics and scattering transforms, for noise levels relevant to Rubin or Euclid. For WL convergence maps with baryonic physics, our models achieve $\sim 2.3$ times stronger constraining power than the power spectrum at these noise levels, also outperforming previous summary statistics. To further explore the possibilities of CNNs for this task, we also discuss transfer learning where we adapt pre-trained models, trained on different tasks or datasets, for cosmological inference, finding that these do not improve the performance.

astro-ph.CO↗

arXiv2024

A field-level emulator for modeling baryonic effects across hydrodynamic simulations

We develop a new and simple method to model baryonic effects at the field level relevant for weak lensing analyses. We analyze thousands of state-of-the-art hydrodynamic simulations from the CAMELS project, each with different cosmology and strength of feedback, and we find that the cross-correlation coefficient between full hydrodynamic and N-body simulations is very close to 1 down to $k\sim10~h{\rm Mpc}^{-1}$. This suggests that modeling baryonic effects at the field level down to these scales only requires N-body simulations plus a correction to the mode's amplitude given by: $\sqrt{P_{\rm hydro}(k)/P_{\rm nbody}(k)}$. In this paper, we build an emulator for this quantity, using Gaussian processes, that is flexible enough to reproduce results from thousands of hydrodynamic simulations that have different cosmologies, astrophysics, subgrid physics, volumes, resolutions, and at different redshifts. Our emulator is accurate at the percent level and exhibits a range of validation superior to previous studies. This method and our emulator enable field-level simulation-based inference analyses and accounting for baryonic effects in weak lensing analyses.

astro-ph.CO↗

arXiv2023

Multiscale Flow for Robust and Optimal Cosmological Analysis

We propose Multiscale Flow, a generative Normalizing Flow that creates samples and models the field-level likelihood of two-dimensional cosmological data such as weak lensing. Multiscale Flow uses hierarchical decomposition of cosmological fields via a wavelet basis, and then models different wavelet components separately as Normalizing Flows. The log-likelihood of the original cosmological field can be recovered by summing over the log-likelihood of each wavelet term. This decomposition allows us to separate the information from different scales and identify distribution shifts in the data such as unknown scale-dependent systematics. The resulting likelihood analysis can not only identify these types of systematics, but can also be made optimal, in the sense that the Multiscale Flow can learn the full likelihood at the field without any dimensionality reduction. We apply Multiscale Flow to weak lensing mock datasets for cosmological inference, and show that it significantly outperforms traditional summary statistics such as power spectrum and peak counts, as well as novel Machine Learning based summary statistics such as scattering transform and convolutional neural networks. We further show that Multiscale Flow is able to identify distribution shifts not in the training data such as baryonic effects. Finally, we demonstrate that Multiscale Flow can be used to generate realistic samples of weak lensing data.

astro-ph.CO↗

arXiv2022

Deterministic Langevin Monte Carlo with Normalizing Flows for Bayesian Inference

We propose a general purpose Bayesian inference algorithm for expensive likelihoods, replacing the stochastic term in the Langevin equation with a deterministic density gradient term. The particle density is evaluated from the current particle positions using a Normalizing Flow (NF), which is differentiable and has good generalization properties in high dimensions. We take advantage of NF preconditioning and NF based Metropolis-Hastings updates for a faster convergence. We show on various examples that the method is competitive against state of the art sampling methods.

stat.ML↗

arXiv2022

Translation and Rotation Equivariant Normalizing Flow (TRENF) for Optimal Cosmological Analysis

Our universe is homogeneous and isotropic, and its perturbations obey translation and rotation symmetry. In this work we develop Translation and Rotation Equivariant Normalizing Flow (TRENF), a generative Normalizing Flow (NF) model which explicitly incorporates these symmetries, defining the data likelihood via a sequence of Fourier space-based convolutions and pixel-wise nonlinear transforms. TRENF gives direct access to the high dimensional data likelihood p(x|y) as a function of the labels y, such as cosmological parameters. In contrast to traditional analyses based on summary statistics, the NF approach has no loss of information since it preserves the full dimensionality of the data. On Gaussian random fields, the TRENF likelihood agrees well with the analytical expression and saturates the Fisher information content in the labels y. On nonlinear cosmological overdensity fields from N-body simulations, TRENF leads to significant improvements in constraining power over the standard power spectrum summary statistic. TRENF is also a generative model of the data, and we show that TRENF samples agree well with the N-body simulations it trained on, and that the inverse mapping of the data agrees well with a Gaussian white noise both visually and on various summary statistics: when this is perfectly achieved the resulting p(x|y) likelihood analysis becomes optimal. Finally, we develop a generalization of this model that can handle effects that break the symmetry of the data, such as the survey mask, which enables likelihood analysis on data without periodic boundaries.

astro-ph.CO↗

arXiv2021

The LHC Olympics 2020: A Community Challenge for Anomaly Detection in High Energy Physics

A new paradigm for data-driven, model-agnostic new physics searches at colliders is emerging, and aims to leverage recent breakthroughs in anomaly detection and machine learning. In order to develop and benchmark new anomaly detection methods within this framework, it is essential to have standard datasets. To this end, we have created the LHC Olympics 2020, a community challenge accompanied by a set of simulated collider events. Participants in these Olympics have developed their methods using an R&D dataset and then tested them on black boxes: datasets with an unknown anomaly (or not). This paper will review the LHC Olympics 2020 challenge, including an overview of the competition, a description of methods deployed in the competition, lessons learned from the experience, and implications for data analyses with future datasets as well as future colliders.

hep-ph↗

arXiv2020

Unsupervised in-distribution anomaly detection of new physics through conditional density estimation

Anomaly detection is a key application of machine learning, but is generally focused on the detection of outlying samples in the low probability density regions of data. Here we instead present and motivate a method for unsupervised in-distribution anomaly detection using a conditional density estimator, designed to find unique, yet completely unknown, sets of samples residing in high probability density regions. We apply this method towards the detection of new physics in simulated Large Hadron Collider (LHC) particle collisions as part of the 2020 LHC Olympics blind challenge, and show how we detected a new particle appearing in only 0.08% of 1 million collision events. The results we present are our original blind submission to the 2020 LHC Olympics, where it achieved the state-of-the-art performance.

cs.LG↗

arXiv2020

MADLens, a python package for fast and differentiable non-Gaussian lensing simulations

We present MADLens a python package for producing non-Gaussian lensing convergence maps at arbitrary source redshifts with unprecedented precision. MADLens is designed to achieve high accuracy while keeping computational costs as low as possible. A MADLens simulation with only $256^3$ particles produces convergence maps whose power agree with theoretical lensing power spectra up to $L{=}10000$ within the accuracy limits of HaloFit. This is made possible by a combination of a highly parallelizable particle-mesh algorithm, a sub-evolution scheme in the lensing projection, and a machine-learning inspired sharpening step. Further, MADLens is fully differentiable with respect to the initial conditions of the underlying particle-mesh simulations and a number of cosmological parameters. These properties allow MADLens to be used as a forward model in Bayesian inference algorithms that require optimization or derivative-aided sampling. Another use case for MADLens is the production of large, high resolution simulation sets as they are required for training novel deep-learning-based lensing analysis tools. We make the MADLens package publicly available under a Creative Commons License (https://github.com/VMBoehm/MADLens).

astro-ph.CO↗

arXiv2020

Learning effective physical laws for generating cosmological hydrodynamics with Lagrangian Deep Learning

The goal of generative models is to learn the intricate relations between the data to create new simulated data, but current approaches fail in very high dimensions. When the true data generating process is based on physical processes these impose symmetries and constraints, and the generative model can be created by learning an effective description of the underlying physics, which enables scaling of the generative model to very high dimensions. In this work we propose Lagrangian Deep Learning (LDL) for this purpose, applying it to learn outputs of cosmological hydrodynamical simulations. The model uses layers of Lagrangian displacements of particles describing the observables to learn the effective physical laws. The displacements are modeled as the gradient of an effective potential, which explicitly satisfies the translational and rotational invariance. The total number of learned parameters is only of order 10, and they can be viewed as effective theory parameters. We combine N-body solver FastPM with LDL and apply them to a wide range of cosmological outputs, from the dark matter to the stellar maps, gas density and temperature. The computational cost of LDL is nearly four orders of magnitude lower than the full hydrodynamical simulations, yet it outperforms it at the same resolution. We achieve this with only of order 10 layers from the initial conditions to the final output, in contrast to typical cosmological simulations with thousands of time steps. This opens up the possibility of analyzing cosmological observations entirely within this framework, without the need for large dark-matter simulations.

astro-ph.CO↗

arXiv2020

Sliced Iterative Normalizing Flows

We develop an iterative (greedy) deep learning (DL) algorithm which is able to transform an arbitrary probability distribution function (PDF) into the target PDF. The model is based on iterative Optimal Transport of a series of 1D slices, matching on each slice the marginal PDF to the target. The axes of the orthogonal slices are chosen to maximize the PDF difference using Wasserstein distance at each iteration, which enables the algorithm to scale well to high dimensions. As special cases of this algorithm, we introduce two sliced iterative Normalizing Flow (SINF) models, which map from the data to the latent space (GIS) and vice versa (SIG). We show that SIG is able to generate high quality samples of image datasets, which match the GAN benchmarks, while GIS obtains competitive results on density estimation tasks compared to the density trained NFs, and is more stable, faster, and achieves higher $p(x)$ when trained on small training sets. SINF approach deviates significantly from the current DL paradigm, as it is greedy and does not use concepts such as mini-batching, stochastic gradient descent and gradient back-propagation through deep layers.

cs.LG↗

arXiv2019

High mass and halo resolution from fast low resolution simulations

Generating mocks for future sky surveys requires large volumes and high resolutions, which is computationally expensive even for fast simulations. In this work we try to develop numerical schemes to calibrate various halo and matter statistics in fast low resolution simulations compared to high resolution N-body and hydrodynamic simulations. For the halos, we improve the initial condition accuracy and develop a halo finder "relaxed-FOF", where we allow different linking length for different halo mass and velocity dispersions. We show that our relaxed-FoF halo finder improves the common statistics, such as halo bias, halo mass function, halo auto power spectrum in real space and in redshift space, cross correlation coefficient with the reference halo catalog, and halo-matter cross power spectrum. We also incorporate the potential gradient descent (PGD) method into fast simulations to improve the matter distribution at nonlinear scale. By building a lightcone output, we show that the PGD method significantly improves the weak lensing convergence tomographic power spectrum. With these improvements FastPM is comparable to the high resolution full N-body simulation of the same mass resolution, with two orders of magnitude fewer time steps. These techniques can be used to improve the halo and matter statistics of FastPM simulations for mock catalogs of future surveys such as DESI and LSST.

astro-ph.CO↗

arXiv2018

A gradient based method for modeling baryons and matter in halos of fast simulations

Fast N-body PM simulations with a small number of time steps such as FastPM or COLA have been remarkably successful in modeling the galaxy statistics, but their lack of small scale force resolution and long time steps cannot give accurate halo matter profiles or matter power spectrum. High resolution N-body simulations can improve on this, but lack baryonic effects, which can only be properly included in hydro simulations. Here we present a scheme to calibrate the fast simulations to mimic the precision of the hydrodynamic simulations or high resolution N-body simulations. The scheme is based on a gradient descent of either effective gravitational potential, which mimics the short range force, or of effective enthalpy, which mimics gas hydrodynamics and feedback. The scheme is fast and differentiable, and can be incorporated as a post-processing step into any simulation. It gives very good results for the matter power spectrum for several of the baryonic feedback and dark matter simulations, and also gives improved dark matter halo profiles. The scheme is even able to find the large subhalos, and increase the correlation coefficient between the fast simulations and the high resolution N-body or hydro simulations. It can also be used to add baryonic effects to the high resolution N-body simulations. While the method has free parameters that can be calibrated on various simulations, they can also be viewed as astrophysical nuisance parameters describing baryonic effects that can be marginalized over during the data analysis. In this view these parameters can be viewed as an efficient parametrization of baryonic effects.

astro-ph.CO↗